A multi-scale feature fusion scene text detection method based on boundary fitting

Abstract

The application discloses a kind of multi-scale feature fusion scene text detection methods based on boundary fitting, belong to computer vision technical field, especially related to scene text detection method based on deep learning.The boundary fitting module is proposed in the present application, the control point is offset to make the boundary deformation by learning with sequence relationship, and the final detection outer contour is obtained.To ensure that the contour contains the complete text region as much as possible, the detection outer contour is more fitted to the irregular shape of text;For the existence of more large-area text in natural scene, the text form contained therein is various and the size distribution is uneven, the present application increases multi-scale feature fusion module in feature extraction part, obtains the feature information of different size receptive field, and improves the problem of incomplete text detection in natural scene.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a scene text detection method based on deep learning. Background Art

[0002] As a key research topic in practical applications, scene text detection technology has made significant progress over the past period, with the emergence of numerous excellent methods and models. Due to the complexity and variability of scene text, scene text detection methods integrate multiple factors and continuously design a variety of functional modules to assist the network in locating and detecting text of arbitrary shapes, uneven sizes, and distorted visual angles.

[0003] Inspired by object detection methods, several studies have achieved promising results on text images with regularly shaped text instances. These methods primarily consist of stacked convolutions, encoding the input image into feature maps. These feature maps are then fed into a classifier to predict the presence and location of text instances at each spatial location. Inspired by one-stage detection models, Liao et al. proposed a Textboxes network that adapts the SSD network to effectively detect and localize text in natural images by defining default boxes as quadrilaterals with varying aspect ratios. To better cover areas where dense text may be present, the default boxes are adjusted using vertical offsets. Zhou's East network, proposed using a U-shaped design, integrates features from different layers. The features at each spatial location are used to regress a rectangular or quadrilateral bounding box for the underlying text instance. The East network also predicts the angle of the detection box to detect text with multiple orientations. This network has played a significant role in text detection, significantly simplifying the pipeline while enabling efficient real-time inference. Based on the same idea, the method proposed by Ma et al. generates rotated region proposals rather than axis-aligned rectangular regions to accommodate text with arbitrary orientations. Although the above methods have good effects on detecting horizontal multi-directional and regularly shaped text, the research on scene text is still very challenging due to interference factors such as variable text shape, texture scale and lighting, especially the detection problem of irregularly shaped text images in natural scenes.

[0004] Different from the above methods, the method of the present invention does not adopt a method based on detection frame regression, but instead uses pixel classification to determine the predicted detection frame based on segmentation.

[0005] Segmentation-based methods locate text regions through pixel-level predictions to enhance robustness to changes in text shape and angle. The DBNet network proposed by Liao et al. inserts a binarization module into the segmentation network for joint optimization, so that the network can adaptively predict the threshold and probability of each pixel in the image, thereby fully distinguishing between foreground and background pixels. The binarization threshold is learned from the network, and the binarization step is completely added to the network for training together. In this way, the final output is very sensitive to the threshold and probability and can be adaptively adjusted, simplifying post-processing while improving the effect of text detection. However, relying solely on segmentation to detect text makes it difficult to distinguish adjacent text regions. The method proposed by Wang et al. shrinks the text region at different scales and gradually enlarges the detected text region until it collides with other instances. This method can effectively extract and distinguish adjacent text instances in the image.

[0006] Other work has explored different approaches to distinguish and extract adjacent text instances. For example, the TextSnake network considers text to be represented as a series of disks sliding along the centerline of the text, similar to a snake, which corresponds to the direction of movement of the text instance. Using this novel approach, the model learns and predicts local attributes, including the centerline, text / non-text regions, disk radius, and orientation. It then uses the local geometric circle to extract the centerline as an ordered list of points, which it then uses to reconstruct text lines, achieving state-of-the-art performance on several curved datasets. In addition to detection methods based on core region expansion, recent work has attempted to fine-tune the feature extraction process and loss functions. For example, the method proposed by Kim et al. focuses on designing a network architecture to reflect the loss function, thereby maximizing the conditional log-likelihood. This module is independent of post-processing of the output features. The feature extraction component of the PAN method consists of a feature pyramid and a feature fusion module. The feature pyramid is a cascaded U-shaped module that incorporates multi-level information to guide better segmentation. The features obtained by this module are then contracted and combined with the segmented image to generate the final detection bounding box.

[0007] Therefore, scene text detection methods are summarized as follows: (1) Regression-based methods predict multiple candidate frames and select the best performing detection frame through post-processing. (2) Segmentation-based methods use text classification information to obtain a segmentation map, thereby obtaining a predicted detection frame. However, both methods have defects and shortcomings. Regression-based methods often use quadrilateral detection frames, which are not effective for detecting curved text in natural scenes; although segmentation-based methods can better detect text in scenes of arbitrary shapes, they rely too much on the accuracy of the predicted detection frame, resulting in problems such as incomplete edge detection or the inclusion of a lot of background noise. Summary of the Invention

[0008] The purpose of the present invention is to propose a multi-scale feature fusion scene text detection method based on boundary fitting, which is more conducive to the separation and distinction of adjacent dense texts by introducing prior knowledge of distance and method; and to design a text detection framework based on boundary fitting, which includes a boundary fitting module and a multi-scale fusion module, and improves the detection accuracy of the model by adaptively deforming the boundary learning and increasing the receptive field. In order to compare the results fairly and objectively, the method of the present invention is trained using the widely used scene text datasets Total-Text and CTW1500, and the positions of polygonal detection boxes are calibrated in the datasets. After training, the test sets in these two datasets are tested, and through comparison, it is found that the method of the present invention improves the accuracy of scene text detection.

[0009] In order to achieve the above objectives, the technical solution of the present invention is: a multi-scale feature fusion scene text detection method based on boundary fitting, the method is:

[0010] The input image passes through the residual network ResNet and the multi-scale feature fusion module in sequence, and then is divided into two paths. One path passes through a convolution module conv1. The output of the convolution module conv1 and the other path are input into the channel fusion module C. The output of the channel fusion module C passes through the boundary fitting module and the prediction output module.

[0011] The loss function L required to predict candidate boundaries PB For: L PB =L cls +αL dis +L v ;

[0012] Among them, L cls represents the text classification loss function, α represents the distance field loss function weight, L v represents the direction vector loss function, L dis represents the distance field loss function;

[0013]

[0014] D p Indicates the distance between the pixel in the predicted detection box and the boundary, Represents the distance between the pixel in the real text image and the pixel in the real label;

[0015] L v =norm_loss+angle_loss

[0016]

[0017] Among them, norm_loss represents distance loss, angle_loss represents angle loss, Vp represents a vector predicted by the model, and represents a vector actually of the image, w(p) represents a weight matrix inversely proportional to the real area of the text region where the pixel point is located, and T represents the text region;

[0018] Further, the residual network ResNet is used to extract multi-scale features of the input image, and the scales include 1 / 16, 1 / 8, and 1 / 4.

[0019] The fusion method of the multi-scale feature fusion module is as follows:

[0020] Step 1: The 1 / 4 size feature map is up-sampled and convolved to generate a 32-channel image with a scale of twice the original, and the scale is 1 / 2.

[0021] Step 2: The 1 / 16, 1 / 8, 1 / 4, and the generated 1 / 2 scale feature maps are sequentially up-sampled, and then the channel number is adjusted to 32 through 1x1 convolution.

[0022] Step 3: The scale feature maps obtained in step 2 are fused in the channel dimension to generate multi-scale fusion features, and then the channel number of the fused feature map is adjusted to the size of the original input image through 1x1 convolution to obtain shared features.

[0023] Further, the input data of the boundary fitting module is first subjected to dimension conversion, sent into a recurrent neural network, then subjected to dimension conversion again and sent into a convolution layer, and finally output.

[0024] The loss function L of the boundary fitting module is as follows: M

[0025]

[0026] wherein T represents a control point set of all text instances in the image, n represents the number of control points set, (k+i) %n represents that the control points are sequentially calculated in the loss calculation, and ||·||1 represents the L1 norm.

[0027] ​The application introduces prior knowledge and realizes detection accuracy improvement by boundary deformation and multi-scale feature fusion. Specifically, in order to alleviate the problem of poor recognition of close text, the application increases direction knowledge auxiliary pixel classification in the network model; by combining the direction information and distance information, the text pixel point direction and distance can effectively separate the close text content and help to predict the text boundary; in view of the problem that the detection method based on segmentation depends on the contour boundary, the application proposes a boundary fitting module, uses sequence relationship to learn to make the control point deviate to deform the boundary, and obtains the final detection outer contour. To ensure that the contour contains the complete text area as much as possible, the detection outer contour is more fitted to the irregular shape of the text; in view of the fact that there are more large-area texts in natural scenes, the text forms contained therein are various and the size distribution is uneven, the application increases a multi-scale feature fusion module in the feature extraction part to obtain feature information of different size receptive fields and improve the problem of incomplete text detection in natural scenes. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 It is a scene text detection network structure diagram based on boundary fitting multi-scale feature fusion.

[0029] Figure 2 It is a boundary fitting module network structure diagram.

[0030] Figure 3 It is a multi-scale feature fusion module structure diagram.

[0031] Figure 4 It is an experimental structure diagram in the Total-Text data set. DETAILED DESCRIPTION

[0032] The main feature of the application is that the application proposes a scene text detection method based on boundary fitting, which is used to improve the text positioning accuracy. (1) To separate close text, the application introduces direction and distance information as prior knowledge. (2) At the same time, the application analyzes the problem that the detection area cannot contain complete text content or contains too much background noise, and designs two effective modules. One is the boundary fitting module, which makes the detection frame more fitted to the text shape by control point deviation. The other is the multi-scale fusion module, which fuses multi-scale feature map information to increase the network receptive field; (3) a large number of experiments in the Total-Text and CTW1500 data sets prove that the modules proposed in the application can effectively improve the performance of the scene text detection network, and the best performance is obtained in the Total-Text data set, and the competitive performance is obtained in the CTW1500 data set.

[0033] The overall detection network framework is as follows Figure 1As shown, prior knowledge is introduced, and it includes a boundary fitting module and a multi-scale feature fusion module; the overall loss function L is the prior knowledge part L PB And the boundary fitting module part loss function L M The weighted result, L = L PB +λL M ; The following is a detailed description of each detail problem designed in the technical solution of the invention;

[0034] (1) Introducing prior knowledge of distance and direction

[0035] The direction field is a unit two-dimensional vector that represents the direction from a text pixel to the nearest non-text pixel. Text pixels near the edge of text point toward the center of the text. If two text regions are adjacent, the pixels at their edges will point in different directions. Therefore, the direction field can effectively distinguish dense text regions based on the difference in direction and distance between the predicted vector and the true direction vector. At the same time, distance information is generated using the direction field distance information to represent the relative position of text pixels within the text region. By combining direction and distance information with the direction and distance of text pixels, dense text content can be effectively separated and help predict text boundaries.

[0036] Among them, the loss function is designed as follows: when predicting text or non-text, the commonly used cross entropy classification loss function is used as the loss function of text classification, L cls is the classification loss function; for the distance field loss, the L2 norm loss function is used, and the distance field loss generated is L dis .

[0037]

[0038] Among them D p Denotes the distance between the pixel in the predicted detection box and the boundary, D p Represents the distance between the pixel in the real text image and the ground truth, p∈T represents all pixel points p belonging to the text area T. Direction vector loss L v It is divided into direction loss and distance loss, which represent the difference between the angle and distance between the actual predicted vector and the vector in the ground truth respectively:

[0039]

[0040] Where V p represents the vector predicted by the model, and V p represents the actual vector of the image, w(p) represents the weight matrix inversely proportional to the actual area of ​​the text region where the pixel is located, and T represents the text region. The final direction field is composed of two losses: angle and distance.

[0041] LPB = L cls + aL dis + L v

[0042] The loss function L required for calculating the prediction candidate boundary can be calculated through the above steps PB , and the loss of L PB is balanced. The weight a of the distance information loss function is set to 3.

[0043] (2) Designing a boundary fitting module

[0044] By sampling a certain number of control points on the predicted boundary at equal distances, the control points are caused to deviate by learning the sequence relationship to deform the boundary, and the final detection outer contour is obtained, and the network result is as shown in Figure 2 . The contour is ensured to contain the complete text region as much as possible, while reducing the non-text background noise in the detection result, so that the detection outer contour is more fitted to the irregularly shaped text.

[0045] The control points are selected in the boundary for training and learning to approach the control points in the ground truth. The present application adopts a point matching loss similar to calculate the deviation between the control points and the real points to adjust the model and adaptively change the direction. The number of control points selected in the prediction in the network and the control points selected on the real boundary are consistent, and the order is also similar (counterclockwise order). The predicted point set is defined as P = {p1, p2,..., p n}, and the point set on the real label detection frame is defined as The corresponding loss is obtained by comparing the distances between the control points, so the L1 norm is used to define the matching loss between p and , and the formula is:

[0046]

[0047] Where n represents the number of control points set, and (k+i) % k is used in the function to ensure that the control points are sequentially calculated in the loss calculation. The loss calculated by this is , which represents the loss of the control points selected on a polygonal detection frame.

[0048] Because an input image often contains multiple text regions, the total boundary control point loss is defined as L M , where T represents the control point set of all text instances in the image:

[0049]

[0050] (3) Designing a multi-scale fusion module

[0051] The present invention upsamples features of different scales and then fuses them. However, unlike the method of downsampling and then upsampling to fuse features of each scale, the module designed by the present invention is simpler and has smaller computational loss. First, the 1 / 4 size feature map is upsampled and convolved (conv) to generate a 32-channel image with a scale twice the original. Then, the 1 / 16, 1 / 8, 1 / 4 and 1 / 2 scale feature maps are upsampled in turn, and the number of channels is adjusted to 32 through 1×1 convolution. Multi-scale fusion features are generated by fusion in the channel dimension, and then the number of channels of the fused feature map is adjusted to the original output size using 1×1 convolution. Its structure is as follows: Figure 3 As shown in the figure, after sequential feature fusion, this method directly incorporates deep feature map information into the final output feature map, allowing the output map to obtain more high-level feature information. It also amplifies the output of the feature pyramid in the original network by a factor of four, making the output consistent with the original image size. The multi-scale feature fusion module fuses feature map information at different scales during the upsampling process, achieving the effect of expanding the receptive field without adding too many parameters.

[0052] The following are the experimental results of our method. Before presenting the experimental results, let's first describe the experimental setup. We used the Total-Text and CTW1500 datasets for experiments, using precision (P), recall (R), and their harmonic mean (F) as evaluation metrics.

[0053] Table 1 compares the proposed method with existing advanced methods. The experimental data comparison in the table demonstrates that the proposed detection network achieves excellent results on natural scene text datasets by introducing direction and distance information, adding a boundary fitting module, and improving the feature fusion module. Backbone represents the feature extraction network type selected by the network model, and External Dataset represents the pre-training dataset used.

[0054] Compared with existing methods, this model achieves state-of-the-art results and is competitive with cutting-edge methods. On the Total-Text dataset, it achieves 89.7% precision and 83.7% recall; on the CTW1500 dataset, it achieves 86.4% precision and 83.7% harmonic mean results.

[0055] At the same time, through Figure 4 Compared with the visual results in , the method proposed in the present invention can detect text areas more completely, the generated boundaries can better fit the text shape, and can effectively separate adjacent dense text areas.

[0056] Table 1 Experimental data comparison with existing advanced methods on Total-Text and CTW1500 datasets

[0057]

Claims

1. A multi-scale feature fusion scene text detection method based on boundary fitting, the method is: The input image passes through the residual network ResNet and the multi-scale feature fusion module in sequence, and then is divided into two paths. One path passes through a convolution module conv1. The output of the convolution module conv1 and the other path are input into the channel fusion module C. The output of the channel fusion module C passes through the boundary fitting module and the prediction output module. The loss function L required to predict candidate boundaries PB For: L PB =L cls +αL dis +L v ; in, L cls represents the text classification loss function, α represents the distance field loss function weight, L v represents the direction vector loss function, L dis represents the distance field loss function; D p Indicates the distance between the pixel in the predicted detection box and the boundary, Represents the distance between the pixel in the real text image and the pixel in the real label; L v =norm_loss+angle_loss Among them, norm_loss represents distance loss, angle_loss represents angle loss, V p represents the vector predicted by the model, and represents the actual vector of the image, w(p) represents the weight matrix inversely proportional to the actual area of ​​the text region where the pixel is located, and T represents the text region; The input data of the boundary fitting module is first dimensionally transformed and fed into the recurrent neural network, then dimensionally transformed again and fed into the convolutional layer before being output. The loss function L of the boundary fitting module M for: Where T represents the set of control points of all text instances in the image, n represents the number of set control points, (k+i)%k indicates that the control points are calculated sequentially in the loss calculation, and ||·||1 represents the L1 norm.

2. The method for detecting scene text using multi-scale feature fusion based on boundary fitting according to claim 1, wherein: The residual network ResNet is used to extract multi-scale features of the input image, and the scales include: 1 / 16, 1 / 8, and 1 / 4; The fusion method of the multi-scale feature fusion module is: Step 1: Upsample and convolve the 1 / 4 size feature map to generate a 32-channel image with twice the original size, and a scale of 1 / 2; Step 2: Upsample the feature maps to 1 / 16, 1 / 8, 1 / 4, and the generated 1 / 2 scales in turn, and then adjust the number of channels to 32 through 1×1 convolution; Step 3: The feature maps of each scale obtained in step 2 are fused in the channel dimension to generate multi-scale fusion features, and then the number of channels of the fused feature map is adjusted to the size of the original input image using 1×1 convolution to obtain shared features.

Images