Text line recognition method and device, electronic equipment and storage medium

Abstract

The invention discloses a text line recognition method and device and electronic equipment, and belongs to the technical field of character recognition. The method comprises the following steps: performing deep multi-scale feature extraction on a text image to obtain multi-scale image features; performing text line detection based on the multi-scale image features to obtain query features and a position detection result of a text line; on the basis of the text image and the position detection result of the text line, obtaining a local region feature of the detected text line; and performing text recognition by fusing the query features and the local region features to obtain a text recognition result corresponding to the line in the text image. According to the method, the text line recognition accuracy is effectively improved.

Description

Technical Field

[0001] The present application relates to the technical field of text recognition, and in particular to a text line recognition method, device, electronic device, and computer-readable storage medium for complex scenes. Background Art

[0002] Tibetan characters in natural scenes contain rich information and have great research and practical value. Due to the complex background, diverse layout, overlapping text, multi-direction, multi-scale, and diverse font styles of natural scene images, Tibetan text recognition under natural scene conditions faces many challenges. The commonly used Tibetan text line recognition methods in the prior art are divided into two categories: the first category is the method of cascading text line detection and text recognition model; the second category is the end-to-end text recognition method. Among them, the first category of methods uses the output of the detection model as the input of the recognition model. The common regression-based detection method is limited by the post-processing algorithm. When processing adjacent overlapping instances, it is easy to make misjudgments. The segmentation-based method segments the image by pixel classification and cannot judge the overlapping area. Due to the limitations of detection and recognition error accumulation and separate optimization, the first category of methods is insufficient in terms of accuracy and speed when processing complex scene images. The second category of methods needs to compress the image proportionally to adapt to the network input when processing large image sizes. This makes the target text size smaller after compression when the target text size in the image is small, and the text detail features are lost, resulting in a decrease in recognition accuracy.

[0003] It can be seen that the text line recognition method in the prior art still needs to be improved. Summary of the invention

[0004] The embodiments of the present application provide a text line recognition method and device, which help to improve the accuracy of text line recognition.

[0005] In a first aspect, an embodiment of the present application provides a text line recognition method, comprising:

[0006] Perform deep multi-scale feature extraction on text images to obtain multi-scale image features;

[0007] Perform text line detection based on the multi-scale image features to obtain query features and text line position detection results;

[0008] Based on the position detection results of the text image and the text line, acquiring local area features of the detected text line;

[0009] The query feature and the local area feature are integrated to perform text recognition, and a text recognition result corresponding to the text line in the text image is obtained.

[0010] In a second aspect, an embodiment of the present application provides a text line recognition device, comprising:

[0011] An image feature acquisition module is used to perform deep multi-scale feature extraction on text images and obtain multi-scale image features;

[0012] The encoding / decoding and text line detection module is used to perform text line detection based on the multi-scale image features and obtain the query features and text line position detection results;

[0013] A local area feature acquisition module, used for acquiring the local area feature of the detected text line based on the position detection result of the text image and the text line;

[0014] The fusion recognition module is used to fuse the query feature and the local area feature to perform text recognition and obtain the text recognition result corresponding to the text line in the text image.

[0015] In a third aspect, an embodiment of the present application further discloses an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the text line recognition method described in the embodiment of the present application when executing the computer program.

[0016] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the text line recognition method disclosed in the embodiment of the present application.

[0017] The text line recognition method disclosed in the embodiment of the present application obtains multi-scale image features by performing deep multi-scale feature extraction on a text image; performs text line detection based on the multi-scale image features to obtain query features and text line position detection results; obtains local area features of the detected text line based on the text image and the position detection results of the text line; finally, performs text recognition by fusing the query features and the local area features to obtain text recognition results corresponding to the text line in the text image, thereby effectively improving the accuracy of text line recognition.

[0018] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0020] Figure 1 is a flow chart of a text line recognition method disclosed in an embodiment of the present application;

[0021] Figure 2 It is one of the structural diagrams of the text recognition model disclosed in the embodiment of the present application;

[0022] Figure 3 This is the second structural diagram of the text recognition model disclosed in the embodiment of the present application;

[0023] Figure 4 is a schematic diagram of a text line detection result in the text line recognition method disclosed in an embodiment of the present application;

[0024] Figure 5 is based on Figure 4 The text line detection result shown is a schematic diagram of extracting the text line area;

[0025] Figure 6 is a schematic diagram of the structure of a text line recognition device disclosed in an embodiment of the present application;

[0026] Figure 7 A block diagram schematically shows an electronic device for executing the method according to the present application; and

[0027] Figure 8 A storage unit for holding or carrying a program code for implementing the method according to the present application is schematically shown. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0029] Reference Figure 1 , a text line recognition method disclosed in an embodiment of the present application includes: steps 110 to 140.

[0030] In some optional embodiments, each step of the text line recognition method disclosed in the embodiments of the present application can be implemented by a text recognition model. Figure 2 As shown, the text recognition model for implementing the text line recognition method includes: a feature extraction module 202, a text line detection module 204, a recognition feature generation module 206 and a text line recognition module 208. The following is an example of the specific implementation of each step of the text line recognition method disclosed in the embodiment of the present application, combined with the structure of the text recognition model.

[0031] Step 110, performing deep multi-scale feature extraction on the text image to obtain multi-scale image features.

[0032] The text images described in the embodiments of the present application include but are not limited to images of Tibetan text.

[0033] Optionally, the feature extraction module 202 is used to perform deep multi-scale feature extraction on the text image to obtain multi-scale image features.

[0034] In some optional embodiments, the feature extraction module 202 may adopt a ResNet network structure. That is, the text recognition model uses ResNet as the backbone network to extract multi-scale features. Performing deep multi-scale feature extraction on the text image to obtain multi-scale image features includes: performing deep multi-scale feature extraction on the text image to obtain a multi-scale feature map; flattening the multi-scale feature map to obtain a flattened feature map; adding position encoding to the flattened feature map to obtain a multi-scale image feature including position information.

[0035] Taking the image size of the input text recognition model as H×W as an example, the feature extraction module 202 extracts features from the input image to obtain feature maps with sizes of 1 / 8, 1 / 16, and 1 / 32 of the original input image, respectively. The feature maps are then flattened, and position information is added to the feature maps using position encoding to obtain image features with a size of N1×d1, where N1 is d1 is a preset feature dimension.

[0036] In some other optional embodiments, other network structures may be used to implement the feature extraction module 202 for extracting deep multi-scale features of the image input to the text recognition model, which are not listed one by one in the embodiments of the present application. The specific network structure for implementing the feature extraction module 202 is not limited in the embodiments of the present application.

[0037] Step 120: Perform text line detection based on the multi-scale image features to obtain query features and text line position detection results.

[0038] Optionally, the text line detection module 204 is used to perform text line detection based on the multi-scale image features to obtain query features and text line position detection results. The position detection results are used to indicate the positions of text lines included in the image (such as the text image) input to the text recognition model. Optionally, the positions of the text lines can be represented by boundary Bezier curves representing the upper and lower boundaries of the text lines. Bezier curves can be used to express curved text lines, especially for text line recognition in complex scenes with a more accurate description effect.

[0039] In an embodiment of the present application, text line detection is performed based on the multi-scale image features to obtain query features and text line position detection results, including: encoding and decoding based on the multi-scale image features to obtain query features; text line prediction is performed based on the query features to obtain the boundary Bezier curves of each text line in the text image as text line position detection results. In an embodiment of the present application, the text line position detection results include: boundary Bezier curves, and may also include: center Bezier curves.

[0040] Accordingly, the text line detection module 204 includes: an encoder, a curve selection module, a decoder and at least one prediction head. Figure 3 The structure of the text recognition model shown is used to illustrate a specific implementation of the text line detection module 204.

[0041] The encoder can adopt an encoder based on a deformable attention mechanism. The multi-scale image features output by the feature extraction module 202 are used as encoder input. First, the multi-scale image features are further encoded by the encoder to obtain an encoding result; then, the encoding result is sent to the curve selection module to select the top K central Bezier curves with the highest scores. Taking the multi-scale image feature size of the input encoder as N1×d1 as an example, the encoding result can be represented as a feature map of N1×d2, where d2 represents the dimension of the encoding vector in the feature map, and each d2-dimensional encoding vector can represent the features of a specific position in the text image input to the text recognition model.

[0042] The curve selection module may include two groups of fully connected layers. After the feature map output by the encoder is input to the curve selection module, each pixel in the feature map passes through a group of fully connected layers, and the last fully connected layer in the group of fully connected layers outputs the offset of a specified number of Bezier control points. The offset of the Bezier control point is used to determine the Bezier curve. For example, if the dimension of the encoding vector output by the last fully connected layer of the curve selection module is 8, the curve selection module can output the offset of 4 Bezier control points. Assuming p i =(p ix ,p iy)∈[0,1] 2 represents a pixel point on the feature map input to the curve selection module, and its corresponding four Bezier control points are j∈{0,1,2,3}, where

[0043] σ is the sigmoid function (i.e., activation function). Represents the offset of the Bezier control point, which is predicted by the fully connected layer. The central Bezier curve can be determined based on the Bezier control point. In some optional embodiments, the central Bezier curve is used to express the position of the text line, which can not only effectively fit the scene text without overlapping problems, but also achieve the effect of simple annotation. The Bezier curve can accurately fit the position of the curved text line, which helps to improve the accuracy of text line detection.

[0044] On the other hand, the curve selection module uses another set of fully connected layers to output the confidence that the area corresponding to the central Bezier curve is a text line, thereby selecting the top K central Bezier curves with the highest confidence, where K is an integer greater than 1.

[0045] Next, the feature map output by the encoder and the Bezier curve output by the curve selection module are used as inputs of the decoder for decoding processing.

[0046] Optionally, the decoder may be a decoder based on a deformable attention mechanism. The decoder input includes two parts of input, one is the output of the encoder, and the other is the fusion query F q Given the first K central Bezier curves, N2 points are uniformly sampled on each central Bezier curve. The points sampled from the central Bezier curve can cover the characters in the text. Each text image can obtain K×N2×2 coordinate information. After position embedding and full connection layer, K×N2×256 position query P is obtained. q In addition, the initialization can learn the query L q , the location query P q and learnable query F q Add together to get the final fusion query F q , F q =P q +L q After that, the output encoding result of the encoder is combined with the fusion query F q The text recognition model can detect the maximum number of text lines, and N2×256 represents the feature vector size of each text line.

[0047] In some optional embodiments, the text line detection module 204 further includes a plurality of different prediction heads, each of which is used to perform output prediction based on the query feature to obtain a prediction result. Figure 3 As shown, the text line detection module 204 further includes: a character category prediction head, a text line prediction head, a boundary curve prediction head, and a center curve prediction head. The character category prediction head is used to predict the character category in the detected instance based on the query feature; the text line prediction head is used to predict whether the detected instance is a text line based on the query feature; the boundary curve prediction head is used to regress the boundary Bezier curve of the text line based on the query feature; and the center curve prediction head is used to regress the center Bezier curve of the text line based on the query feature.

[0048] Optionally, the character category prediction head may be composed of a fully connected layer, the text line prediction head may be composed of a fully connected layer, and the boundary curve prediction head and the center curve prediction head may be composed of multiple fully connected layers. In the embodiments of the present application, there is no limitation on the specific implementation of the character category prediction head, the text line prediction head, the boundary curve prediction head, and the center curve prediction head.

[0049] Among them, the specific implementation methods of the encoder, decoder, curve selection module and each prediction head refer to the prior art and will not be repeated in the embodiments of the present application.

[0050] Step 130 : acquiring local area features of the detected text line based on the position detection results of the text image and the text line.

[0051] In some embodiments of the present application, the local area features of the detected text line are obtained based on the position detection results of the text image and the text line by the recognition feature generation module 206. The recognition feature generation module 206 can be constructed by a convolutional neural network and a recurrent neural network.

[0052] As described above, the position detection result of the text line includes: a central Bezier curve and a boundary Bezier curve. Optionally, the local area features of the detected text line are obtained based on the position detection results of the text image and the text line, including: extracting a text line area of ​​a preset size from the text image based on the boundary Bezier curve; performing feature extraction on the text line area to obtain a first text area feature; extracting the temporal dependency between the first text area features to obtain the local area features of the text line.

[0053] For example, first, according to the boundary Bezier curves of the upper boundary and the lower boundary of each text line output by the text line detection module, a text line with a size of h is determined from the text image input to the text recognition model. align × align As a text line area, the image area is used as the text line area. For example, BezierAlign (Bezier curve alignment) technology can be used, with the text image and the boundary Bezier curve of the text line as input. BezierAlign will uniformly sample an image area of ​​a preset size from the text image according to the input information as the text line area. Figure 4 Taking the text image shown in FIG. 40 and the boundary Bezier curves shown in 402 and 404 as examples, the image region of a preset size extracted by using the BezierAlign technology is as follows: Figure 5 As shown, h is uniformly sampled from the curved image region between the two boundary Bezier edges. align × align The sampling points obtained by sampling constitute the text line area.

[0054] Next, each text line region is input into the convolutional neural network for feature extraction to obtain the first text area feature of each text line region; then, the first text area feature of each text line region is passed to the recurrent neural network, and the temporal dependency relationship in the first text area feature of each text line region is captured by the recurrent neural network, and finally the local area feature of the corresponding text line is obtained. The specific network structure of the convolutional neural network and the recurrent neural network can be found in the prior art and will not be repeated here.

[0055] In the embodiment of the present application, by using BezierAlign to extract the text line area in the original image, it can adapt to text lines of any shape, and the text line features extracted by technologies such as ROIAlign (region of interest alignment) are more accurate. On the other hand, when processing large image sizes and small target text fonts, more text detail information can be saved by extracting local area features of the text line from the original image input to the text recognition model, thereby improving the accuracy of text line recognition.

[0056] Step 140: fusing the query feature and the local area feature to perform text recognition, and obtaining a text recognition result corresponding to the text line in the text image.

[0057] Next, the query features of the text lines extracted in the above steps and the local area features of each text line are integrated to perform text recognition, thereby obtaining text recognition results corresponding to each text line in the text image.

[0058] In some optional embodiments, the fusion of the query feature and the local area feature to perform text recognition and obtain the text recognition result corresponding to the text line in the text image includes: weighted fusion of the query feature and the local area feature to obtain a fused feature; and character classification prediction based on the fused feature to obtain the text recognition result corresponding to the text line in the text image.

[0059] In some optional embodiments, the text line recognition module fuses the query features and the local area features to perform text recognition, and obtains the text recognition result corresponding to the text line in the text image. Optionally, the text line recognition module includes: a dimension alignment layer, a normalization layer, a splicing layer, a weight prediction layer, a weighted addition layer, etc. For example, the local area features and the query features can be weightedly fused in sequence by dimension alignment, normalization layer, splicing layer, weight prediction layer, weighted addition layer, etc. to obtain a fused feature, and then the fused feature is sent to a prediction head composed of a fully connected layer for character classification to obtain a character classification output result of the text line area.

[0060] In some optional embodiments, the weighted fusion of the query feature and the local area feature to obtain a fused feature includes: dimensionally aligning the query feature and the local area feature to obtain an aligned query feature and an aligned local area feature, respectively; normalizing the aligned query feature and the aligned local area feature to obtain a normalized query feature and a normalized local area feature, respectively; splicing the normalized query feature and the normalized local area feature to obtain a spliced ​​feature; based on the spliced ​​feature, predicting a first weight corresponding to the normalized query feature and a second weight corresponding to the normalized local area feature; multiplying the first weight by the normalized query feature to obtain a weighted query feature, and multiplying the second weight by the normalized local area feature to obtain a weighted local area feature; and using the feature sum obtained by adding the weighted query feature and the weighted local area feature as a fused feature.

[0061] The first weight and the second weight are dynamic weights predicted by the text line recognition module of the pre-trained text recognition model according to the spliced ​​features of the currently input normalized query features and the normalized local area features.

[0062] For example, the query feature and the local area feature are sent to the fully connected layer for feature dimension alignment to obtain the aligned query feature F query and align local region features F local , then, align the query feature F query and align local region features F localThe features are normalized through the normalization layer to obtain the normalized query feature F′ query And the normalized local region feature F l ' ocal ; Then, normalize the query feature F′ query And the normalized local region feature F l ' ocal Input to the concatenation layer, concatenated in the feature dimension to obtain the concatenated feature F concat ; The splicing feature F concat Send to the fully connected layer to predict the weights a of the two features f , the first weight corresponding to the normalized query feature and the second weight corresponding to the normalized local area feature are obtained, which are recorded as: a f [0] and a f [1]; The predicted first weight a f [0] and the second weight a f [1] After multiplying the corresponding normalized features of respectively, we get the weighted local area features and weighted query features, and then add the two together to get the final fusion feature F fusion Optionally, Formula F can be used fusion =a f [0]×F′ query +a f [1]×F l ' ocal The fusion features are calculated.

[0063] Optionally, the weight prediction formula is as follows: a f =softmax(W a [F′ query ; F l ' ocal ]), where W a It represents the weight prediction parameter of the text line recognition module, and softmax() represents the activation function.

[0064] Finally, the fusion feature F fusion Input to the fully connected layer for classification mapping, and you can get the character classification output result corresponding to the text line.

[0065] Furthermore, the character classification output is transcribed through CTC (Connectionist Temporal Classification) to solve the problem of learning sequences of indefinite length, and the output is decoded into the final predicted character sequence to obtain the text line recognition result.

[0066] The text recognition model is pre-trained. That is, the deep multi-scale feature extraction of the text image and the acquisition of the multi-scale image features also include: training the text recognition model based on the acquired training samples, wherein the text recognition model includes: a feature extraction module, a text line detection module, a recognition feature generation module and a text line recognition module.

[0067] Optionally, the training samples include: synthetic samples. Using synthetic samples to train the text recognition model can make the features extracted by the text recognition model have strong representation and generalization capabilities.

[0068] In order to make this solution clearer, the training process of the text recognition model is explained below.

[0069] Optionally, the training text recognition model includes: performing deep multi-scale feature extraction on the sample image of the training sample through the feature extraction module to obtain multi-scale image features; based on the multi-scale image features, performing encoding and decoding processing through the text line detection module to obtain query features, and performing multi-task prediction on the query features to obtain prediction results corresponding to each task, wherein the prediction results include: text line classification prediction results, character classification prediction results, and text line position detection results; obtaining local area features of the detected text lines based on the sample image and the position detection results of the text lines through the recognition feature generation module; performing text recognition by fusing the query features and the local area features through the text line recognition module to obtain text recognition results corresponding to the text lines in the text image; obtaining the model loss of the text recognition model based on the encoding loss and decoding loss of the text line detection module and the text line recognition result prediction loss of the text line recognition module, and iteratively training the text recognition model with the goal of optimizing the model loss.

[0070] The text line detection module is used to perform deep multi-scale feature extraction on the sample image. The specific implementation method for obtaining multi-scale image features is described above and will not be repeated here.

[0071] The specific implementation method of encoding and decoding the multi-scale image features to obtain the query features is described above and will not be repeated here.

[0072] In the embodiments of the present application, a multi-task joint training method is used to train the text recognition model. In the model training stage, for the query features output by the decoder of the text line detection module, multiple prediction heads in the text recognition model are further used to perform different prediction tasks respectively to obtain corresponding outputs. For example, the query features are used as inputs of the character category prediction head, the text line prediction head, the boundary curve prediction head and the center curve prediction head respectively, and different prediction results are obtained through different prediction heads. The specific implementation of each prediction head can refer to the prior art and will not be repeated in the embodiments of the present application.

[0073] Optionally, the position detection result includes: a boundary Bezier curve and a center Bezier curve of a text line. The character category of the instance detected by the character category prediction head is used as a character classification prediction result; the probability that the instance detected by the text line prediction head is a text line is used as a text line classification prediction result; the boundary curve prediction head predicts the boundary curve of the text line to obtain a boundary Bezier curve; the center curve prediction head predicts the center curve of the text line to obtain a center Bezier curve.

[0074] The above prediction results can be used to directly calculate the loss function of the model and / or used as input for subsequent steps.

[0075] The specific implementation method of obtaining the local area features of the detected text line based on the position detection results of the sample image and the text line by the recognition feature generation module can be found in the relevant description of the reasoning stage above, which will not be repeated here.

[0076] The specific implementation method of fusing the query feature and the local area feature by the text line recognition module to perform text recognition and obtain the text recognition result corresponding to the text line in the sample image can be found in the relevant description of the reasoning stage above, which will not be repeated here.

[0077] After obtaining the prediction results output by the text line detection module and the text line recognition module, further, in the process of processing the sample image according to the text recognition model until the above prediction results are output, the output of the specified step and the sample label corresponding to the sample image are calculated, and the model loss can be completed by optimizing the model loss.

[0078] In an embodiment of the present application, the encoding loss and decoding loss of the text line detection module and the text line detection result prediction loss of the text line recognition module are integrated to calculate the model loss of the text recognition model.

[0079] The following is an example of how to calculate the model loss.

[0080] Referring to the query-based target detection method in the prior art, the Hungarian algorithm is first used to calculate the label set G and the prediction set The optimal allocation of Minimize the matching loss. The label set is the label set of the sample images in the training sample; the prediction set is the text line position prediction result and recognition result obtained after performing the above multi-level feature extraction, feature encoding, decoding processing, and text line prediction operations on the sample images in the training sample. The optimal allocation can be calculated by the following formula:

[0081]

[0082] Where M is the number of text line instances in the sample image, L match Represents the matching loss calculation formula, which can be used to calculate the matching loss of the encoder and the matching loss of the decoder respectively.

[0083] Among them, the matching loss function of the text line detection module encoder is as follows:

[0084]

[0085] The matching loss function of the text line detection module decoder is as follows:

[0086]

[0087] Among them, λ cls_enc , pos_enc , cls_dec , pos_dec , char_dec is a hyperparameter, indicating the weight of each loss; L cls Indicates the use of focal loss with an increased balancing factor to calculate the text line instance classification loss; L pos represents L1distance loss, which is used to calculate the regression coordinate loss of the Bezier curve; L char Represents CTC loss, which is used to calculate character classification loss.

[0088] Next, based on the optimal allocation of the label set and the prediction set, the model loss is calculated. Optionally, the model loss of the text recognition model can be expressed as: L all =L enc +L dec +L recg , where L all represents the model loss, L enc represents the coding loss, L dec represents the decoding loss, L recgRepresents the text recognition loss. Various types of losses can be calculated by the following formula:

[0089]

[0090] Among them, L represents the number of feature points, K represents the number of queries, l represents the lth feature point, k represents the kth query, and λ′ cls_enc ,λ′ cpos_enc ,λ′ cls_dec ,λ′ cpos_dec ,λ′ tbpos_dec ,λ′ char_dec ,λ′ char_recg is a hyperparameter.

[0091] Used below replace Represents the text classification loss, using Unified replacement The specific calculation method is as follows:

[0092]

[0093] Among them, 1 represents the indicator function, Im(μ) represents the image of mapping μ, Indicates that the prediction target of the *th feature point or query is the text instance probability.

[0094]

[0095] Among them, cNum represents the number of sampling points on the central Bezier curve. Represents the coordinates of the nth sampling point on the central Bezier curve of the *th feature point or the corresponding sample label of the query. Represents the coordinates of the nth sampling point on the central Bezier curve of the *th feature point or query prediction.

[0096]

[0097] Among them, tpNum represents the number of sampling points on the boundary Bezier curve. Indicates the coordinates of the nth sampling point on the upper curve of the sample label corresponding to the kth query, represents the coordinates of the nth sampling point on the upper curve of the kth query prediction, Indicates the coordinates of the nth sampling point of the lower curve corresponding to the kth query label, Represents the coordinates of the nth sampling point on the lower curve of the kth query prediction.

[0098]

[0099] in, represents the sample label text corresponding to the k-th query, represents the predicted text of the k-th query and CTC() represents the transcription function.

[0100] In an embodiment of the present application, text line detection is regarded as a set prediction problem, and the sample labels of the text lines and the prediction results are matched one-to-one. Each sample label of each text line has a query corresponding to it. The number of queries is fixed and will not generate a large number of duplicate targets. A certain degree of target overlap problem can be solved without the need for post-processing.

[0101] On the other hand, the joint training of text line detection and text recognition jointly optimizes the parameters of the deep multi-scale feature extraction network, so that the detection network can obtain high-level text features and thus have more accurate text line detection capabilities, ultimately improving the accuracy of text line recognition.

[0102] In summary, the text line recognition method disclosed in the embodiments of the present application obtains multi-scale image features by performing deep multi-scale feature extraction on the text image; performs text line detection based on the multi-scale image features to obtain query features and text line position detection results; obtains local area features of the detected text line based on the text image and the position detection results of the text line; finally, text recognition is performed by fusing the query features and the local area features to obtain text recognition results corresponding to the text lines in the text image, thereby effectively improving the accuracy of text line recognition.

[0103] Reference Figure 6 The present application also discloses a text line recognition device, which includes:

[0104] An image feature acquisition module 610 is used to perform deep multi-scale feature extraction on text images to obtain multi-scale image features;

[0105] The encoding, decoding and text line detection module 620 is used to perform text line detection based on the multi-scale image features and obtain query features and text line position detection results;

[0106] A local area feature acquisition module 630, configured to acquire local area features of a detected text line based on the text image and the position detection results of the text line;

[0107] The fusion recognition module 640 is used to fuse the query feature and the local area feature to perform text recognition and obtain the text recognition result corresponding to the text line in the text image.

[0108] Optionally, before performing deep multi-scale feature extraction on the text image and obtaining multi-scale image features, the device further includes: a model training module,

[0109] The model training module is used to train a text recognition model based on the acquired training samples, wherein the text recognition model includes: a feature extraction module, a text line detection module, a recognition feature generation module and a text line recognition module, and the training text recognition model includes:

[0110] Performing deep multi-scale feature extraction on the sample image of the training sample by the feature extraction module to obtain multi-scale image features;

[0111] Based on the multi-scale image features, encoding and decoding are performed by the text line detection module to obtain query features, and multi-task prediction is performed on the query features to obtain prediction results corresponding to each task, wherein the prediction results include: position detection results;

[0112] Acquire, by the recognition feature generation module, local area features of the detected text line based on the sample image and the position detection result of the text line;

[0113] The text line recognition module fuses the query feature and the local area feature to perform text recognition, and obtains a text recognition result corresponding to the text line in the text image;

[0114] Based on the encoding loss and decoding loss of the text line detection module and the text line recognition result prediction loss of the text line recognition module, the model loss of the text recognition model is obtained, and the text recognition model is iteratively trained with the goal of optimizing the model loss.

[0115] Optionally, the position detection result of the text line includes: a boundary Bezier curve, and the local area feature acquisition module 630 is further used to:

[0116] Based on the boundary Bezier curve, extracting a text line area of ​​a preset size from the text image;

[0117] Extracting features from the text line region to obtain first text region features;

[0118] The temporal dependency relationship between the first text region features is extracted to obtain the local region features of the text line.

[0119] Optionally, the fusing the query feature and the local area feature to perform text recognition to obtain a text recognition result corresponding to a text line in the text image includes:

[0120] Performing weighted fusion on the query feature and the local area feature to obtain a fused feature;

[0121] Character classification prediction is performed based on the fusion features to obtain text recognition results corresponding to the text lines in the text image.

[0122] Optionally, performing weighted fusion on the query feature and the local area feature to obtain a fused feature includes:

[0123] Performing dimension alignment processing on the query feature and the local area feature to obtain an aligned query feature and an aligned local area feature respectively;

[0124] Normalizing the aligned query feature and the aligned local area feature respectively to obtain a normalized query feature and a normalized local area feature;

[0125] Splicing the normalized query feature and the normalized local area feature to obtain a spliced ​​feature;

[0126] Based on the concatenated feature, predicting a first weight corresponding to the normalized query feature and a second weight corresponding to the normalized local area feature;

[0127] multiplying the first weight by the normalized query feature to obtain a weighted query feature, and multiplying the second weight by the normalized local area feature to obtain a weighted local area feature;

[0128] The feature sum obtained by adding the weighted query feature and the weighted local area feature is used as the fusion feature.

[0129] Optionally, the encoding, decoding and text line detection module 620 is further configured to:

[0130] Perform encoding and decoding processing based on the multi-scale image features to obtain query features;

[0131] The text line prediction is performed based on the query feature, and the boundary Bezier curve of each text line in the text image is obtained as the position detection result of the text line.

[0132] The text line recognition device disclosed in the embodiment of the present application is used to implement the text line recognition method described in the embodiment of the present application. The specific implementation methods of each module of the device will not be repeated here, and reference can be made to the specific implementation methods of the corresponding steps in the method embodiment.

[0133] The text line recognition device disclosed in the embodiment of the present application obtains multi-scale image features by performing deep multi-scale feature extraction on a text image; performs text line detection based on the multi-scale image features to obtain query features and text line position detection results; obtains local area features of the detected text line based on the text image and the position detection results of the text line; finally, performs text recognition by fusing the query features and the local area features to obtain text recognition results corresponding to the text line in the text image, thereby effectively improving the accuracy of text line recognition.

[0134] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0135] The above is a detailed introduction to a text line recognition method and device provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

[0136] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0137] The various component embodiments of the present application can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all functions of some or all components in the electronic device according to the embodiment of the present application. The present application can also be implemented as a device or apparatus program (e.g., computer program and computer program product) for executing part or all of the methods described herein. Such a program implementing the present application can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0138] For example, Figure 7 An electronic device that can implement the method according to the present application is shown. The electronic device can be a PC, a mobile terminal, a personal digital assistant, a tablet computer, etc. The electronic device traditionally includes a processor 710 and a memory 720 and a program code 730 stored on the memory 720 and can be run on the processor 710, and the processor 710 implements the method described in the above embodiment when executing the program code 730. The memory 720 can be a computer program product or a computer-readable medium. The memory 720 can be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk or a ROM. The memory 720 has a storage space 7201 for the program code 730 of the computer program for executing any method step in the above method. For example, the storage space 7201 for the program code 730 can include various computer programs for implementing the various steps in the above method respectively. The program code 530 is a computer-readable code. These computer programs can be read from one or more computer program products or written into the one or more computer program products. These computer program products include program code carriers such as hard disks, compact disks (CDs), memory cards or floppy disks. The computer program includes a computer readable code, and when the computer readable code is executed on an electronic device, the electronic device is caused to execute the method according to the above embodiment.

[0139] The embodiment of the present application further discloses a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the text line recognition method as described in the embodiment of the present application are implemented.

[0140] Such a computer program product may be a computer-readable storage medium, which may have a computer-readable storage medium having a Figure 7The memory 720 in the electronic device shown in the figure is similar to the storage segment, storage space, etc. The program code can be compressed and stored in the computer-readable storage medium in an appropriate form. The computer-readable storage medium is generally as shown in the reference Figure 8 The portable or fixed storage unit. Generally, the storage unit includes computer readable code 730', which is a code read by a processor. When the code is executed by the processor, each step in the method described above is implemented.

[0141] The term "one embodiment", "embodiment" or "one or more embodiments" herein means that a particular feature, structure or characteristic described in conjunction with the embodiment is included in at least one embodiment of the present application. In addition, please note that the examples of the term "in one embodiment" here do not necessarily all refer to the same embodiment.

[0142] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0143] In the claims, any reference signs placed between brackets shall not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present application may be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third etc. does not indicate any order. These words may be interpreted as names.

[0144] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A text line recognition method, characterized in that: The method comprises: Perform deep multi-scale feature extraction on text images to obtain multi-scale image features; Perform text line detection based on the multi-scale image features to obtain query features and text line position detection results; Based on the position detection results of the text image and the text line, acquiring local area features of the detected text line; The query feature and the local area feature are integrated to perform text recognition, and a text recognition result corresponding to the text line in the text image is obtained.

2. The method according to claim 1, characterized in that The deep multi-scale feature extraction of the text image and obtaining the multi-scale image features also include: Based on the acquired training samples, a text recognition model is trained, wherein the text recognition model includes: a feature extraction module, a text line detection module, a recognition feature generation module and a text line recognition module, and the training text recognition model includes: Performing deep multi-scale feature extraction on the sample image of the training sample by the feature extraction module to obtain multi-scale image features; Based on the multi-scale image features, encoding and decoding are performed by the text line detection module to obtain query features, and multi-task prediction is performed on the query features to obtain prediction results corresponding to each task, wherein the prediction results include: position detection results; Acquire, by the recognition feature generation module, local area features of the detected text line based on the sample image and the position detection result of the text line; The text line recognition module fuses the query feature and the local area feature to perform text recognition, and obtains a text recognition result corresponding to the text line in the text image; Based on the encoding loss and decoding loss of the text line detection module and the text line recognition result prediction loss of the text line recognition module, the model loss of the text recognition model is obtained, and the text recognition model is iteratively trained with the goal of optimizing the model loss.

3. The method according to claim 1, characterized in that The position detection result of the text line includes: a boundary Bezier curve, and acquiring the local area feature of the detected text line based on the text image and the position detection result of the text line includes: Based on the boundary Bezier curve, extracting a text line area of ​​a preset size from the text image; Extracting features from the text line region to obtain first text region features; The temporal dependency relationship between the first text region features is extracted to obtain the local region features of the text line.

4. The method according to claim 1, characterized in that: The fusing the query feature and the local area feature to perform text recognition and obtain a text recognition result corresponding to a text line in the text image includes: Performing weighted fusion on the query feature and the local area feature to obtain a fused feature; Character classification prediction is performed based on the fusion features to obtain text recognition results corresponding to the text lines in the text image.

5. The method according to claim 4, characterized in that The weighted fusion of the query feature and the local area feature to obtain the fused feature includes: Performing dimension alignment processing on the query feature and the local area feature to obtain an aligned query feature and an aligned local area feature respectively; Normalizing the aligned query feature and the aligned local area feature respectively to obtain a normalized query feature and a normalized local area feature; Splicing the normalized query feature and the normalized local area feature to obtain a spliced ​​feature; Based on the concatenated feature, predicting a first weight corresponding to the normalized query feature and a second weight corresponding to the normalized local area feature; multiplying the first weight by the normalized query feature to obtain a weighted query feature, and multiplying the second weight by the normalized local area feature to obtain a weighted local area feature; The feature sum obtained by adding the weighted query feature and the weighted local area feature is used as the fusion feature.

6. The method according to claim 1, characterized in that The performing text line detection based on the multi-scale image features to obtain query features and text line position detection results includes: Perform encoding and decoding processing based on the multi-scale image features to obtain query features; The text line prediction is performed based on the query feature, and the boundary Bezier curve of each text line in the text image is obtained as the position detection result of the text line.

7. A text line recognition device, characterized in that: The device comprises: An image feature acquisition module is used to perform deep multi-scale feature extraction on text images and obtain multi-scale image features; The encoding / decoding and text line detection module is used to perform text line detection based on the multi-scale image features and obtain the query features and text line position detection results; A local area feature acquisition module, used for acquiring the local area feature of the detected text line based on the position detection result of the text image and the text line; The fusion recognition module is used to fuse the query feature and the local area feature to perform text recognition and obtain the text recognition result corresponding to the text line in the text image.

8. The device according to claim 7, characterized in that The encoding / decoding and text line detection module is further used for: Perform encoding and decoding processing based on the multi-scale image features to obtain query features; The text line prediction is performed based on the query feature, and the boundary Bezier curve of each text line in the text image is obtained as the position detection result of the text line.

9. An electronic device comprising a memory, a processor, and a program code stored in the memory and executable on the processor, characterized in that: When the processor executes the program code, the method according to any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium having program code stored thereon, characterized in that: When the program code is executed by a processor, the steps of the method described in any one of claims 1 to 6 are implemented.

Images