An end-to-end method for text detection and recognition based on deep learning

Abstract

The application discloses an end-to-end method for text detection and recognition based on deep learning, comprising the following steps: constructing and training an end-to-end model based on a neural network, and using the trained end-to-end model to detect and recognize document text in a given image.The end-to-end model for text detection and recognition has the following advantages: in the skeleton network stage, the designed fusion module achieves cross-scale feature fusion, which not only reduces the calculation cost, but also improves the accuracy of the model; in the optimization stage of training, the loss function of the text detection task and the loss function of the text recognition task are combined, the loss of the text detection and recognition is considered, the text detection and recognition task is optimized, and the time cost caused by training is reduced; in the inference stage, the model only needs to be forward-inferred once, and the text box, the corresponding confidence and the text content can be obtained at a low cost.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to an end-to-end method for text detection and recognition based on deep learning. BACKGROUND

[0002] Scene text detection and recognition is an important branch of target detection and a popular research field of computer vision, and is widely applied to the scenes of document recognition, dial plate recognition, blind assistance, real-time translation and the like. Text detection and text recognition are inseparable, text detection is the basis of text recognition, and text recognition is the embodiment of the value of text detection. At present, the in-depth research on the two tasks of text detection and text recognition separately makes the two technologies achieve good results. However, the two separate tasks cannot apply the two technologies to specific scenes, and the end-to-end method for text detection and text recognition becomes a research path for the organic combination of the two technologies.

[0003] Due to the complexity of the text, there are dramatic scale transformations in the text, presenting diversity in font, color, shape, direction, and language, which increases the difficulty of text detection and recognition tasks. In order to solve the above difficulties, deep learning technology is applied to solve the above difficulties, and has become the main way to solve it. Strong deep learning models often have the characteristics of large model and slow inference, which affects the practicability of the model. The most widely cited text detection method based on segmentation is DBNet(Minghui Liao, Zhaoyi Wan, Cong Yao, Kai Chen, and Xiang Bai. Real-time scene text detection with differentiable binarization. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 11474–11481, 2020.), through experiments it can be determined that the backbone network in this method occupies the vast majority of parameter quantity and most of the computational quantity, the feature fusion module occupies most of the computational quantity, the above two modules also occupy 45% and 39% in time consumption. Considering that the text detected by the model usually only occupies a small part of the image, especially for street datasets such as ICDAR2015(Dimosthenis Karatzas, Lluis Gomez-Bigorda, Anguelos Nicolaou, Suman Ghosh, Andrew Bagdanov, Masakazu Iwamura, Jiri Matas, Lukas Neumann, Vijay Ramaseshan Chandrasekhar, Shijian Lu, et al. Icdar 2015 competition on robust reading. In 2015 13th international conference on document analysis and recognition (ICDAR), pages 1156–1160. IEEE, 2015.) FPN top-down and bottom-up design aims to fully understand the features of the entire image, but the text box occupies a small proportion and the shape is simple.The currently widely used text recognition method is a CRNN method (B. Shi, X. Bai, and C. Yao, "An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognition.") which is improved by applying the CTC loss in the voice field to the text recognition, and good recognition effect is obtained. The above text detection and text recognition are two independent models, and for practical application, the combination of detection and recognition is really valuable. SUMMARY

[0004] In order to more accurately and efficiently detect and recognize text, solve the landing of text detection and text recognition and the combination of the two, the present application proposes an end-to-end method for text detection and recognition based on deep learning, in order to improve the usability of the text detection and text recognition model, the method is realized by reducing the size of the model and speeding up the inference speed, and combining the two models into an end-to-end model for text detection and recognition.

[0005] The object of the present application is achieved at least by one of the following technical solutions.

[0006] An end-to-end method for text detection and recognition based on deep learning, comprising the following steps:

[0007] S1, constructing and training an end-to-end model for text detection and recognition based on neural network;

[0008] S2, using the trained end-to-end model for text detection and recognition to detect and recognize the text in the image to be detected.

[0009] Further, step S1 comprises the following steps:

[0010] S1.1, constructing an asymmetric double-branch feature extraction network, the double-branch feature extraction network is respectively used to extract local information and global information, and an adaptive weighting fusion module is used to combine the information of the double branches to obtain a feature map;

[0011] S1.2, using a differentiable binarization method to obtain probability information and threshold information in the feature map, and combining the probability information and the threshold information in the feature map into binary information for distinguishing text regions and non-text regions, and taking the probability information, the threshold information and the binary information as the output of the detection branch;

[0012] S1.3, in the training mode, obtaining the text box from the training label; in the inference mode, obtaining the text box from the binary information; according to the text box, cutting the corresponding region from the feature map, using the perspective projection mode to obtain the rectangular region feature with fixed height as the input feature of the text recognition task;

[0013] S1.4, inputting the region feature of the text recognition into the sequence decoder composed of LSTM to obtain the text information related to the text recognition region as the output of the recognition branch;

[0014] S1.5, calculating the loss of the detection branch and the recognition branch, taking the weighted sum of the loss of the detection branch and the loss of the recognition branch as the loss function, and taking the weighted loss as the loss of the end-to-end model for training.

[0015] Further, in step S1.1, the branch for extracting local information includes 3 sub-modules, each sub-module includes a convolution layer with a stride of 2, two convolution layers with a stride of 1 and a filter size of 3x3, and a Relu activation function is followed after each sub-module as output, to quickly obtain shallow features at 1 / 8 scale; the branch for extracting global information uses the stem block module in yolo5, and 1 / 32 scale deep features are obtained through 5 times of 1 / 2 downsampling.

[0016] Further, the adaptive weighted fusion module includes the following steps:

[0017] 1) for the shallow features and deep features with input scales of HxW, H / 4xW / 4 and channel number C, four groups of filters are used, each group of filters has a size of 1 and a number of C / 2; the shallow features and the deep features are convolved twice to obtain two shallow feature maps and two deep feature maps with a channel number reduced to C / 2;

[0018] 2) change the dimensions of the two shallow feature maps and the two deep feature maps, combine the length and width into one dimension, form a two-dimensional matrix with the channel number, the number of rows of the matrix is the channel number C / 2, the number of columns of the shallow feature matrix is (HxW), and the number of columns of the deep feature matrix is (H / 4xW / 4);

[0019] 3) take one shallow feature matrix and one deep feature matrix from step 2), transpose one of the matrices to make its column number C / 2, and multiply the transposed matrix by the other matrix to obtain a two-dimensional matrix of (H / 4xW / 4)x(HxW) as a weighted fusion matrix;

[0020] 4) Take another matrix of shallow features and another matrix of deep features different from step 3) from step 2), divide the weighted fusion matrix in step 3) by (HxW) and perform matrix multiplication with the shallow feature matrix to obtain a shallow weight matrix with column number (H / 4xW / 4) and row number C / 2; divide the weighted fusion matrix in step 3) by (H / 4xW / 4) and perform matrix multiplication with the deep feature matrix to obtain a deep weight matrix with column number (HxW) and row number C / 2;

[0021] 5) Restore the shallow weight matrix to a three-dimensional graph of H / 4xW / 4xC, add it to the deep features at the input end, and perform an average pooling layer operation to obtain a gating weight of 1x1xC;

[0022] 6) Restore the deep weight matrix to a three-dimensional graph of HxWxC, add it to the shallow features at the input end to obtain a feature map of HxWxC;

[0023] 7) Multiply the feature map in step 6) by the gating weight to output a weighted feature map.

[0024] Further, in step S1.2, a convolutional layer with a step size of 1 and a filter size of 3x3 is used to convolve the features obtained in step S1.1 at a scale of 1 / 8 to obtain a feature map with a channel number of 2 at a scale of 1 / 8; for each pixel, the information in the two channels respectively represents the probability that the pixel belongs to the text region and the threshold value used to classify the pixel in the graph as text or non-text, the single-channel image containing the probability information is taken as a probability map, the single-channel image containing the threshold value information is taken as a threshold value map, and the probability map and the threshold value map are subjected to a differentiable binarization operation to obtain a binary map, which is regarded as the third channel of the feature map; for the probability image pixel p p and the threshold value image pixel p t and the binary image pixel p b The differentiable binarization is calculated using the following method:

[0025]

[0026] Further, in step S1.4, a bidirectional LSTM is used as a sequence decoder.

[0027] Further, the height of the input is fixed at 1 and the width is of any length, and the output length is fixed at 1024, and the number of channels is the same as the number of classes of characters.

[0028] Further, in step S1.5, the following loss function is used to calculate the loss:

[0029] L = L rec + aL probably + bL threshold + Lbinary

[0030] wherein L, L rec , L probably , L threshold and L binary are total loss, recognition loss, probability map loss, threshold map loss and binary map loss respectively, and alpha and beta are weight coefficients for balancing the probability map loss and the threshold map loss.

[0031] Further, the recognition task uses CTC loss as the loss, the probability map uses BCE loss as the loss, the threshold map uses L1 loss as the loss, and the binary map uses Dice loss as the loss.

[0032] Further, the step S2 comprises the following steps:

[0033] S2.1, inputting an image to be detected, using a detection branch in the trained end-to-end model to perform text detection, and obtaining a text binary map and an image feature;

[0034] S2.2, using a findcontours function of cv2 to obtain a text box, and inputting the text box and the image feature into perspective projection to obtain a corrected text rectangular feature;

[0035] S2.3, inputting the text rectangular feature into a recognition area to obtain a recognition result corresponding to the text box;

[0036] S2.4, outputting the text box, a corresponding confidence and recognized text content.

[0037] Compared with the prior art, the present application has the following advantages and technical effects:

[0038] (1) The present application effectively accelerates the model inference speed by using a double-branch feature extraction network. At the same time, the accuracy and recall rate of text detection do not decrease.

[0039] (2) To accelerate the model inference speed, the text detection and text recognition are combined into an end-to-end model, and the features extracted once are applied to the two tasks, reducing repeated calculation, and achieving obvious effects in model lightweight and model acceleration.

[0040] (3) The method of perspective projection is used to perform more accurate area extraction at the output end of text detection. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 is a flow chart of an end-to-end method for text detection and recognition based on deep learning in the embodiments of the present application;

[0042] Figure 2This is a diagram illustrating the end-to-end method model architecture for text detection and recognition based on deep learning, as described in an embodiment of the present invention.

[0043] Figure 3 This is a comparison diagram showing the effects of using projection and not using projection methods in an embodiment of the present invention. Detailed Implementation

[0044] To make the technical solutions and advantages of the present invention clearer, the specific implementation of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments, but the implementation and protection of the present invention are not limited thereto.

[0045] like Figure 1 , Figure 2 The end-to-end method for text detection and recognition based on deep learning, as shown, includes the following steps:

[0046] S1. Construct and train an end-to-end model for text detection and recognition based on neural networks;

[0047] S1.1 Construct an asymmetric two-branch feature extraction network (AFF-Net), with two branches for extracting local and global information respectively, and use an adaptive weighted fusion module to combine the information from the two branches to obtain feature maps.

[0048] In one embodiment, the branch for extracting local information is divided into three sub-modules. Each sub-module consists of one convolutional layer with a stride of 2 and two convolutional layers with a stride of 1, each with a filter size of 3×3. Each sub-module is followed by a ReLU activation function as its output to quickly obtain shallow features at a 1 / 8 scale. The branch for extracting global information uses the stem block module from YOLOv5, obtaining deep features at a 1 / 32 scale through five 1 / 2 downsampling operations. The adaptive weighted fusion module includes the following steps:

[0049] 1) For shallow and deep features with input scales of H×W and H / 4×W / 4 respectively, and C channels, four sets of filters are used, each with a size of 1 and a number of filters of C / 2. Two convolutions are performed on the shallow and deep features to obtain two shallow feature maps and two deep feature maps with the number of channels reduced to C / 2.

[0050] 2) Change the dimensions of the two shallow feature maps and the two deep feature maps, merge the length and width into one dimension, and form a two-dimensional matrix together with the number of channels. The number of rows in the matrix is ​​C / 2 of the number of channels. The number of columns in the shallow feature matrix is ​​(H×W), and the number of columns in the deep feature matrix is ​​(H / 4×W / 4).

[0051] 3) Obtain a matrix of shallow features and a matrix of deep features from 2), transpose one of the matrices so that the number of columns is C / 2. Multiply the transposed matrix with the other matrix to obtain a two-dimensional matrix of size (H / 4 x W / 4) x (H x W) as the weighted fusion matrix;

[0052] 4) Obtain another matrix of shallow features and another matrix of deep features from 2) which are different from 3). Divide the weighted fusion matrix in 3) by (H x W) and multiply it with the matrix of shallow features to obtain a shallow weight matrix of size (H / 4 x W / 4) x (C / 2). Divide the weighted fusion matrix in 3) by (H / 4 x W / 4) and multiply it with the matrix of deep features to obtain a deep weight matrix of size (H x W) x (C / 2);

[0053] 5) Restore the shallow weight matrix to a three-dimensional map of size H / 4 x W / 4 x C, add it with the deep features at the input end, and pass it through an average pooling layer to obtain a gating weight of size 1 x 1 x C;

[0054] 6) Restore the deep weight matrix to a three-dimensional map of size H x W x C, add it with the shallow features at the input end to obtain a feature map of size H x W x C;

[0055] 7) Multiply the feature map in step 6) with the gating weight to output a weighted feature map.

[0056] S1.2, obtain the probability information and threshold information in the feature map using the method of differentiable binarization, and combine the two into binary information for distinguishing the text region from the non-text region, and take the probability information, threshold information, and binary information as the output of the detection branch;

[0057] This step uses a convolutional layer with a step size of 1 and a filter size of 3 x 3 to convolve the feature at the scale of 1 / 8 obtained in S1.1 to obtain a feature map at the scale of 1 / 8 with 2 channels. For each pixel, the information in the two channels respectively represents the probability that the pixel belongs to the text region. The single-channel image composed of the probability information can be regarded as a probability map, and the single-channel image composed of the threshold information can be regarded as a threshold map. Perform differentiable binarization operation on the probability map and the threshold map to obtain a binary map, which is regarded as the third channel of the feature map; for the probability map pixel p p and the threshold map pixel p t and the binary map pixel p b The differentiable binarization is calculated using the following method:

[0058]

[0059] S1.3, in the training mode, obtaining the text box from the training label; in the inference mode, obtaining the text box from the binary information. According to the text box, the corresponding region is intercepted from the feature map, and a rectangular region feature with a fixed height is obtained by using perspective projection as the input feature of the text recognition task;

[0060] S1.4, inputting the region feature of the text recognition into the sequence decoder composed of LSTM to obtain the text information related to the text recognition region as the output of the recognition branch;

[0061] As one of the embodiments, the bidirectional LSTM is used as the sequence decoder, the input height is fixed to 1, the width is arbitrary length, the output length is fixed to 1024, and the number of channels is the same as the number of categories of characters.

[0062] S1.5, calculating the loss of the detection branch and the recognition branch, taking the weighted sum of the loss of the detection branch and the loss of the recognition branch as the loss function, and training the end-to-end model with the above weighted sum as the loss of this training.

[0063] As one of the embodiments, the following loss function is used to calculate the loss:

[0064] L=L rec +αL probably +βL threshold +L binary

[0065] Wherein L, L rec , L probably , L threshold and L binary are the total loss, the recognition loss, the probability map loss, the threshold map loss and the binary map loss respectively, and α and β are the weight coefficients for balancing the probability map loss and the threshold map loss.

[0066] The recognition task uses CTC loss as the loss, the probability map uses BCE loss as the loss, the threshold map uses L1 loss as the loss, and the binary map uses Dice loss as the loss.

[0067] S1.6, repeated model training is performed to obtain appropriate results.

[0068] S2, using the trained end-to-end model of text detection and recognition to detect and recognize the text in the given image.

[0069] S2.1, inputting the image to be detected, using the text detection branch in the trained end-to-end model to detect the text, and obtaining the text binary map and the image feature;

[0070] S2.2, using the findcontours function of cv2 to obtain the text box, and sending the text box and image features into perspective projection to obtain the corrected text rectangle features;

[0071] S2.3, sending the rectangle features into the recognition area to obtain the recognition result corresponding to the text box;

[0072] S2.4, outputting the text box, corresponding confidence and recognized text content.

[0073] In the embodiment, the text detection method of the present application shows good lightweight effect. Compared with the detection model DBNet, the text detection method of the present application reduces the parameter quantity from 12.935M to 1.995M; when the input picture size is 640x640, the calculation quantity is reduced from 46.819G to 12.127G; the number of pictures processed per second (FPS) is increased from 33.51 to 47.05; in the verification on the ICDAR2015 dataset, the F1-Score of the method is the same as that of the DBNet method, both being 82.25%; in the CTW1500 dataset, the F1-Score of the method is slightly improved compared with that of the DBNet, from 81.04% to 81.06%, ensuring that the method maintains the detection effect while being lightweight. In the end-to-end connection effect, the perspective projection adopted by the method also shows good effect. Figure 3 The text detection and recognition of a small ticket are shown, the upper right corner of each text detection box is marked with the result of text recognition alignment and the detection confidence, and in the same detection box, the projection connection end-to-end method shows stronger text recognition ability.

[0074] The preferred embodiments of the present application disclosed above are only used to help explain the present application. The preferred embodiments do not describe all the details, nor limit the present application to the specific embodiments described. Obviously, according to the content of the present application, many modifications and changes can be made. The present application selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present application, so that those skilled in the art can well understand and use the present application. The present application is limited only by the claims and their entire scope and equivalents.

Claims

1. An end-to-end method of text detection and recognition based on deep learning, characterized in that, Comprise the following steps: S1, construct and train the end-to-end model of text detection and recognition based on neural network; Comprise the following steps: S1.1, construct an asymmetric double-branch feature extraction network, and the double-branch feature extraction network is respectively used for extracting local information and global information, and an adaptive weighted fusion module is used to combine the information of the double branches to obtain a feature map; S1.2, use the method of differentiable binarization to obtain the probability information and threshold information in the feature map, and combine the probability information and threshold information in the feature map into binary information for distinguishing text regions and non-text regions, and the probability information, threshold information and binary information are taken as the output of the detection branch; S1.3, in the training mode, obtain the text box from the training label; In the inference mode, obtain the text box from the binary information; According to the text box, the corresponding region is intercepted from the feature map, and the rectangular region feature with fixed height is obtained by using perspective projection as the input feature of the text recognition task; S1.4, input the rectangular region feature of text recognition into the sequence decoder composed of LSTM to obtain the text information related to the text recognition region as the output of the recognition branch; S1.5, calculate the loss of the detection branch and the recognition branch, take the weighted sum of the loss of the detection branch and the loss of the recognition branch as the loss function, and take the weighted loss as the loss for training the end-to-end model; S2, use the trained end-to-end model of text detection and recognition to detect and recognize the text in the image to be detected.

2. The end-to-end method for text detection and recognition based on deep learning according to claim 1, characterized in that, In step S1.1, the branch for extracting local information includes three sub-modules, each sub-module includes one convolutional layer with a stride of 2, two convolutional layers with a stride of 1 and a filter size of 3x3, and a Relu activation function is followed after each sub-module as output, to quickly obtain shallow features at 1 / 8 scale; The branch for extracting global information uses the stem block module in yolov5 to obtain deep features at 1 / 32 scale through 5 times of 1 / 2 downsampling. 3.The end-to-end method of text detection and recognition based on deep learning according to claim 1, wherein, The adaptive weighted fusion module comprises the following steps: 1) for the input size of , , the shallow feature and the deep feature with the channel number of , four groups of filters are used, each group of filter has a size of 1 and a number of ; the shallow feature and the deep feature are convolved twice to obtain two shallow feature maps and two deep feature maps with the channel number reduced to ; 2) change the dimension of two shallow feature maps and two deep feature maps, combine the length and width into one dimension, and form a two-dimensional matrix with the number of channels, the number of rows of the matrix is the number of channels , the number of rows of the matrix of shallow features is , and the number of rows of the matrix of deep features is ; 3) take a matrix of shallow features and a matrix of deep features from step 2), transpose one of the matrices so that the number of columns is , right multiply the other matrix to obtain a two-dimensional matrix of size ( ) x ( ) as a weighted fusion matrix; 4) take another matrix of another shallow feature different from step 3) and another matrix of another deep feature from step 2), divide the weighted fusion matrix in step 3) by and then perform matrix multiplication with the shallow feature matrix to obtain a shallow weight matrix with column number and row number ; divide the weighted fusion matrix in step 3) by and then perform matrix multiplication with the deep feature matrix to obtain a deep weight matrix with column number and row number ; 5) the shallow weight matrix is restored to a three-dimensional graph, which is added to the deep features of the input end, and the average pooling layer operation is performed to obtain the gating weight; 6) restore the deep weight matrix to a three-dimensional graph, add the shallow features of the input end to get the feature map of ; 7) multiply the feature map of step 6) by the gating weight to output the weighted feature map.

4. The end-to-end method for text detection and recognition based on deep learning according to claim 1, characterized in that, In step S1.2, a convolutional layer with a step of 1 and a filter size of 3x3 is used to convolve the features at 1 / 8 scale obtained in step S1.1 to obtain a feature map with 2 channels at 1 / 8 scale; For each pixel, the information in the two channels represents the probability that the pixel belongs to the text region and the threshold value used to classify the pixel in the image as text or non-text, respectively. A single-channel image containing probability information is taken as a probability map, and a single-channel image containing threshold information is taken as a threshold map. The probability map and the threshold map are subjected to a differentiable binarization operation to obtain a binary map, which is regarded as the third channel of the feature map; For a probability image pixel and a threshold image pixel and a binary image pixel Differential binarization can be computed using the following method: 。 5. The end-to-end method for text detection and recognition based on deep learning according to claim 1, wherein, In step S1.4, a bidirectional LSTM is used as a sequence decoder.

6. The end-to-end method for text detection and recognition based on deep learning according to claim 5, wherein, The input height is fixed at 1 and the width is of any length, the output length is fixed at 1024, and the number of channels is the same as the number of character categories.

7. The end-to-end method for text detection and recognition based on deep learning according to claim 1, wherein, In step S1.5, the following loss function is used for loss calculation: wherein , , and are the total loss, the identification loss, the probability map loss, the threshold map loss and the binary map loss, respectively, and a, b are the weight coefficients balancing the probability map loss and the threshold map loss, respectively.

8. The end-to-end method for text detection and recognition based on deep learning according to claim 7, wherein, The recognition task uses CTCloss as the loss, the probability map uses BCEloss as the loss, the threshold map uses L1 loss as the loss, and the binary map uses Dice loss as the loss.

9. The end-to-end method for text detection and recognition based on deep learning according to any one of claims 1-8, characterized in that, Step S2 comprises the following steps: S2.1, input the image to be detected, use the detection branch in the trained end-to-end model to perform text detection, and obtain a text binary map and an image feature; S2.2, use the findcontours function of cv2 to obtain a text box, and input the text box and the image feature into perspective projection to obtain a corrected text rectangular feature; S2.3, input the text rectangular feature into a recognition area to obtain a recognition result corresponding to the text box; S2.4, output the text box, the corresponding confidence, and the recognized text content.

Images