A scene text detection method based on deep learning

Abstract

The present invention belongs to the field of artificial intelligence technology, and proposes a scene text detection method based on deep learning. The present invention first constructs a basic feature extraction network module, and then constructs a dual-weight guided flow alignment module, which is combined with a feature pyramid network module to handle the multi-scale change problem in text detection; then a dual-path symmetric residual module is used to enhance edge features to obtain a segmentation result; finally, a text box is generated from the segmentation map. The present invention realizes a fast and lightweight text detection model, which can not only detect text targets of any shape in complex scenes, but also has accurate detection results and a certain guarantee for the detection speed; it has extremely strong application value.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a scene text detection method based on deep learning. Background Art

[0002] In recent years, deep learning technology has been continuously developing, and its application in the field of text detection has become more and more extensive, such as autonomous driving, scene understanding, and product identification. The core task of the text detection algorithm is to distinguish text instances from background areas. There are usually great differences in text in natural scenes. For example, there may be differences in text size, font, and text color in pictures. The text in such images is more complex than the text in traditional electronic documents, and the accuracy of detection using traditional text detection methods is relatively low. At present, due to the extensive application of deep learning technology, many scholars have set their sights on the detection of scene text images. There are currently two main scene text detection methods based on deep learning, one is a regression-based text detection method, and the other is a segmentation-based text detection method.

[0003] The regression-based text detection method mentioned in "Faster R-CNN: Towards real-time object detection with region proposal networks" and "Ssd: Single shot multibox detector" aims to regress the location coordinates of the text target to be detected. This method regards the text line as a detection target and uses the target detection network to directly detect the coordinates of the text line. The regression-based method can be divided into indirect regression and direct regression methods according to whether the anchor is preset. The regression-based method cannot handle tilted text and long text well. The segmentation-based detection method proposed in "TextSnake: A flexible representation for detecting text of arbitrary shapes" and "Textfield: Learning a deep direction field for irregular scene text detection" obtains the text region segmentation map through network detection, and then obtains the text box by post-processing. Since the current text datasets are usually annotated in the form of text lines, the use of segmentation detection methods will cause irrelevant pixels between characters to be annotated as characters during training, resulting in adjacent text lines sticking together in the segmentation results. Therefore, the advantage of the segmentation method is that its segmentation results contain information such as the direction, position, and length of the text, and there is no need to consider the characteristics of long texts and deformed texts, but its initial segmentation results are prone to adhesion. Summary of the invention

[0004] The technical problem to be solved by the present invention is to make up for the shortcomings of current scene text detection accuracy and propose a scene text detection method based on deep learning to achieve the purpose of high accuracy and high detection rate.

[0005] The technical solution of the present invention:

[0006] A scene text detection method based on deep learning includes the following steps:

[0007] Step 1: Construct a feature extraction network;

[0008] The feature extraction network includes a basic feature extraction network module, a dual-weight-guided stream alignment module, and a feature pyramid network module;

[0009] The basic feature extraction network module uses the ResNet or MobileNet network structure as the backbone. The image is input to different network layers, and the output features of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the image size are extracted respectively. The number of channels corresponding to different network layers is 64, 128, 256, and 512 respectively.

[0010] The input of the dual-weight guided flow alignment module is divided into F low and F high Two parts, F low Represents the output features of the current basic feature extraction network layer, F high represents the output features of the previous level double weight guided flow alignment module; first, F high Upsample and use bilinear interpolation to get the same value as F low The two feature maps are channel-concatenated and convoluted once to obtain a semantic flow field. At the same time, the channel-concatenated feature maps are sequentially convoluted by 3×3 convolution, Relu activation function, and 1×1 convolution to obtain a spatial weight map. The obtained semantic flow field is compared with F high Through the optical flow mapping operation, this operation can well solve the misalignment problem that occurs when using bilinear interpolation for upsampling. The aligned feature map is multiplied by the spatial weight map to obtain a high-resolution feature map rich in spatial semantic information; at the same time, F high The channel weight map obtained after the channel attention mechanism module in the dual-weight guided stream alignment module is consistent with F low Multiply them together to obtain a high-resolution feature map rich in channel information; multiply the high-resolution feature map rich in spatial semantic information, the high-resolution feature map rich in channel information, and the initial F low The feature map obtained by adding the three together is used as the output of the double-weight-guided stream alignment module;

[0011] The dual-weight guided flow alignment module is reused three times, and the feature pyramid network module is used to fuse the output of the dual-weight guided flow alignment module; first, the 1 / 32 size feature map C5 output by the basic feature extraction network module is feature expanded using the atrous spatial convolutional pooling pyramid network (ASPP) to obtain a 1 / 32 size feature map P5, and the 1 / 16 size feature map C4 and the 1 / 32 size feature map P5 output by the basic feature extraction network module are input into the dual-weight guided flow alignment module; then the dual-weight guided flow The output features of the alignment module are added to the 1 / 16-size feature map C4 output by the basic feature extraction network module to obtain a 1 / 16-size feature map P4; repeat the above steps to obtain a 1 / 8-size feature map P3 and a 1 / 4-size feature map P2; finally, the 1 / 32-size feature map P5, the 1 / 16-size feature map P4, and the 1 / 8-size feature map P3 are upsampled to the same size as the 1 / 4-size feature map P2, and then through channel cascading, finally a feature map F1 with a size of 1 / 4 of the original image and a channel number of 256 is obtained;

[0012] Step 2: Input the feature map F1 into the dual-path asymmetric residual module for edge enhancement to obtain the feature map F2; the enhancement of edge features is inseparable from local features, local context features, and global context features.

[0013] The dual-path asymmetric residual module includes a feature pyramid convolution layer, an asymmetric convolution layer, and a point-by-point convolution layer. After the feature map F1 is input into the feature pyramid convolution layer for feature extraction, the number of channels is halved and transmitted to two different branches. The two different branches respectively use 3×1 and 1×3 depth convolutions with expansion rates and 1×3 and 3×1 depth convolutions to alleviate the contradiction between classification and positioning and promote the aggregation of information. The receptive field is increased without reducing the resolution of the feature map, thereby capturing more information. The number of channels is restored at the end of both branches through a point-by-point convolution layer with a convolution kernel of 1×1. After the feature maps output by the two branches are added, the channels are shuffled to promote the exchange of information, and finally added to the input feature map of the module to promote the fusion of information, and the feature map F2 is obtained.

[0014] Step 3: The probability map P and the threshold map T are predicted from the feature map F2; the approximate binary map B is calculated by the probability map P and the threshold map T. The calculation formula is as follows:

[0015]

[0016] Among them: B i,j is the generated approximate binary image; P i,j is a probability graph, T i,j is the threshold map; k is the magnification; the experimental value is 50.

[0017] Step 4: Generate a text box

[0018] First, a fixed threshold is set to convert the probability map into an approximate binary map; the approximate binary map consists of points with pixel values ​​0 and 1, where points with a pixel value of 1 belong to the text area, and vice versa, they belong to the non-text area; then the contour of the approximate binary map is calculated, each contour is traversed, and the predictions of the text box that are less than the set minimum length are removed; the enclosing rectangle is calculated for each contour, and then the prediction score of the rectangle is calculated, and the rectangle is reversely shrunk to obtain the true rectangle size; finally, it is restored to the original image size.

[0019] This method is implemented based on the Pytorch framework. Two GPU graphics cards (Nvidia 1080Ti) are used for training and testing in the experiment. The stochastic gradient descent algorithm is used for optimization, the training batch size is 16, the initial learning rate is set to 0.001, and the iterative training is 2000 epochs. The data enhancement methods used are: random angle rotation, random region cropping, random flipping, and all images are resized to 640×640.

[0020] Beneficial effects of the present invention: The present invention makes full use of the semantic segmentation algorithm's strong ability to distinguish between foreground and background areas, and combines the dual-weight-guided flow alignment module with the feature pyramid network module to perform multi-scale feature extraction, thereby improving the ability to detect multi-size text in images. Edge enhancement is performed through a dual-path asymmetric residual module, making the predicted probability map and threshold map more accurate, thereby improving the accuracy of the entire text detection algorithm, and having extremely strong application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is the entire network structure diagram. The flow alignment is a double-weight guided flow alignment module.

[0022] Figure 2 It is a dual-weight guided flow alignment module, ⊙ represents pixel-by-pixel multiplication;

[0023] Figure 3 It is a two-way asymmetric residual module. DETAILED DESCRIPTION

[0024] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings and technical solutions.

[0025] A scene text detection method based on deep learning, the steps are as follows:

[0026] (1) Constructing basic feature extraction network module

[0027] The basic feature extraction network uses the ResNet18 network structure as the backbone, and the input is a 3-channel RGB image, such as Figure 1As shown in the figure, C2, C3, C4, and C5 represent the 4th, 6th, 9th, and 13th layer features in ResNet18, which are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the input image size, respectively, and the corresponding number of channels are 64, 128, 256, and 512, respectively.

[0028] (2) Constructing a dual-weight guided flow alignment module

[0029] like Figure 2 As shown, the input of this module is F low and F high Two parts, F low Represents the output features of the current layer of the feature advance network, F high represents the output features of the previous level stream alignment module. First, high Upsample and use bilinear interpolation to get the same value as F low The two feature maps are then channel-concatenated and convolved once more to predict the semantic flow field. At the same time, the channel-concatenated feature maps are sequentially convolved by 3×3 convolution, Relu function activation, and 1×1 convolution to obtain the spatial weight map. The obtained semantic flow field is compared with F high After the optical flow mapping operation, it is multiplied by the spatial weight map to obtain a high-resolution feature map rich in spatial semantic information. high The channel weight map obtained after the channel attention mechanism module (CAM) is consistent with F low Multiply them together to obtain a high-resolution feature map rich in channel information; multiply the high-resolution feature map rich in spatial semantic information, the high-resolution feature map rich in channel information, and the initial F low The feature map obtained by adding the three together is used as the output of the double-weight-guided stream alignment module;

[0030] (3) Constructing feature pyramid network module

[0031] The feature pyramid network module can fuse the output of the dual-weight-guided stream alignment module. Figure 1As shown in the figure, the dual-weight guided stream alignment module is reused three times in the feature pyramid network. First, the 1 / 32-size feature map C5 output by the basic feature extraction network module is feature expanded using the atrous spatial convolutional pooling pyramid network (ASPP) to obtain a 1 / 32-size feature map P5. The 1 / 16-size feature maps C4 and P5 output by the basic feature extraction network module are used as the input of the dual-weight guided stream alignment module. Then, the dual-weight guided stream alignment module output is added to C4 to obtain a 1 / 16-size feature map P4. Repeat the above steps to obtain a feature map P3 with a size of 1 / 8 and a feature map P2 with a size of 1 / 4. Finally, P5, P4, and P3 are upsampled to the same size as P2, and then through channel cascading, a feature map with a size of 1 / 4 of the original image and a number of channels of 256 is obtained.

[0032] (4) Constructing a dual-path asymmetric residual module

[0033] An asymmetric residual module is constructed to enhance the feature map after channel cascading, which plays a role in edge enhancement. The enhancement of edge features is inseparable from local features, local context features, and global context features. Figure 3 As shown in the figure, the dual-path asymmetric residual module starts to extract features from the input feature map using the feature pyramid convolution layer, and then reduces the number of channels by half and sends it to two different branches. The upper and lower branches can effectively alleviate the contradiction between classification and positioning, and the asymmetric convolution can effectively promote the aggregation of information. The lower branch uses deep convolution with kernels of 1×3 and 3×1, and the upper branch uses deep convolution with dilation rate of 3×1 and 1×3, which increases the receptive field without reducing the resolution of the feature map, thereby capturing more information. At the end of the two branches, point-by-point convolution with a kernel of 1×1 is used to restore the number of channels. After the feature maps of the two branches are added, the channels are shuffled to promote the exchange of information. Finally, they are added to the input feature map of the module to promote information fusion, forming a residual structure.

[0034] (5) Get an approximate binary image

[0035] The feature map obtained by the two-way asymmetric residual module is used to predict the probability map P and the threshold map T. The approximate binary map is obtained by differentiable binarization. The calculation formula is as follows:

[0036]

[0037] in:

[0038] B i,j is the generated approximate binary image;

[0039] P i,j is a probability graph, T i,j is the threshold map;

[0040] k is the magnification factor, and the experimental value is 50.

[0041] (6) Generate text box

[0042] First, a fixed threshold of 0.3 is set, and the probability map is converted into an approximate binary map through the fixed threshold. The approximate binary map consists of points with pixel values ​​0 and 1, where points with pixel value 1 belong to the text area, and vice versa. Then, the contour of the approximate binary map is calculated, and each contour is traversed to remove predictions that are too small, that is, a minimum length of the text box is initially set. If the predicted length is less than this length, it is directly discarded; the enclosing rectangle is calculated for each contour, and then the prediction score of the rectangle is calculated, and the rectangle is reversely shrunk to obtain the true rectangle size; finally, it is restored to the original image size.

[0043] (7) Training methods

[0044] This method is implemented based on the Pytorch framework. Two GPU graphics cards (Nvidia 1080Ti) are used for training and testing in the experiment. The stochastic gradient descent algorithm is used for optimization, the training batch size is 16, the initial learning rate is set to 0.001, and the iterative training is 2000 epochs. The data enhancement methods used are: random angle rotation, random region cropping, random flipping, and all images are resized to 640×640.

Claims

1. A scene text detection method based on deep learning, characterized in that: The steps include: Step 1: Construct a feature extraction network; The feature extraction network includes a basic feature extraction network module, a dual-weight-guided stream alignment module, and a feature pyramid network module; The basic feature extraction network module uses the ResNet or MobileNet network structure as the backbone. The image is input to different network layers, and the output features of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the image size are extracted respectively. The number of channels corresponding to different network layers is 64, 128, 256, and 512 respectively. The input of the dual-weight guided flow alignment module is divided into F low and F high Two parts, F low Represents the output features of the current basic feature extraction network layer, F high represents the output features of the previous level double weight guided flow alignment module; first, F high Upsample and use bilinear interpolation to get the same value as F low The two feature maps are channel-concatenated and convoluted once to obtain a semantic flow field. At the same time, the channel-concatenated feature maps are sequentially convoluted by 3×3 convolution, Relu activation function, and 1×1 convolution to obtain a spatial weight map. The obtained semantic flow field is compared with F high The optical flow mapping operation is used to solve the misalignment problem when upsampling using bilinear interpolation. The aligned feature map is multiplied by the spatial weight map to obtain a high-resolution feature map rich in spatial semantic information. high The channel weight map obtained after the channel attention mechanism module in the dual-weight guided stream alignment module is consistent with F low Multiply them together to obtain a high-resolution feature map rich in channel information; multiply the high-resolution feature map rich in spatial semantic information, the high-resolution feature map rich in channel information, and the initial F low The feature map obtained by adding the three together is used as the output of the double-weight-guided stream alignment module; The dual-weight guided flow alignment module is reused three times, and the feature pyramid network module is used to fuse the output of the dual-weight guided flow alignment module; first, the 1 / 32 size feature map (C5) output by the basic feature extraction network module is feature expanded using the dilated spatial convolutional pooling pyramid network to obtain a 1 / 32 size feature map (P5), and the 1 / 16 size feature map (C4) and the 1 / 32 size feature map (P5) output by the basic feature extraction network module are input into the dual-weight guided flow alignment module; then the output of the dual-weight guided flow alignment module is The feature map (C4) of size 1 / 16 output by the feature extraction network module is added to obtain a feature map (P4) of size 1 / 16; the above steps are repeated to obtain a feature map (P3) of size 1 / 8 and a feature map (P2) of size 1 / 4; finally, the feature map (P5) of size 1 / 32, the feature map (P4) of size 1 / 16, and the feature map (P3) of size 1 / 8 are upsampled to the same size as the feature map (P2) of size 1 / 4, and then through channel cascading, finally a feature map F1 with a size of 1 / 4 of the original image and a channel number of 256 is obtained; Step 2: Input the feature map F1 into the dual-path asymmetric residual module for edge enhancement to obtain the feature map F2; The two-way asymmetric residual module includes a feature pyramid convolution layer, an asymmetric convolution layer and a point-by-point convolution layer. After the feature map F1 is input into the feature pyramid convolution layer for feature extraction, the number of channels is halved and transmitted to two different branches. The two different branches respectively use 3×1 and 1×3 depth convolutions with expansion rates and 1×3 and 3×1 depth convolutions to alleviate the contradiction between classification and positioning and promote the aggregation of information. At the end of both branches, the number of channels is restored by a point-by-point convolution layer with a convolution kernel of 1×1. After the feature maps output by the two branches are added, the channels are shuffled to promote the exchange of information, and finally added to the input feature map of the module to promote the fusion of information, and the feature map F2 is obtained. Step 3: The probability map P and the threshold map T are predicted from the feature map F2; the approximate binary map B is calculated by the probability map P and the threshold map T. The calculation formula is as follows: Among them: B i,j is the generated approximate binary image; P i,j is a probability graph, T i,j is the threshold map; k is the magnification; Step 4: Generate a text box First, a fixed threshold is set to convert the probability map into an approximate binary map; the approximate binary map consists of points with pixel values ​​0 and 1, where points with a pixel value of 1 belong to the text area, and vice versa, they belong to the non-text area; then the contour of the approximate binary map is calculated, each contour is traversed, and the predictions of the text box that are less than the set minimum length are removed; the enclosing rectangle is calculated for each contour, and then the prediction score of the rectangle is calculated, and the rectangle is reversely shrunk to obtain the true rectangle size; finally, it is restored to the original image size.

Images