Text recognition method, device, electronic device and storage medium

Abstract

The present application discloses a text recognition method, device, and electronic device, and belongs to the field of text recognition technology. The method includes: a feature extraction module of a pre-trained text recognition model performs deep multi-scale feature extraction on a text image to obtain multi-scale image features; an encoder of the text recognition model performs encoding processing on the multi-scale image features to obtain encoding features; an area selection module of the text recognition model performs text line area selection processing on the encoding features to obtain a first query vector corresponding to the text line; a decoder of the text recognition model performs layer-by-layer decoding processing based on the first query vector and the encoding features to obtain a second query vector; and a plurality of different first prediction heads of the text recognition model predict text-related information on the second query vector to obtain a text information prediction result. This method combines multimodal features such as vision and semantics for decoding, thereby improving the accuracy of text recognition.

Description

Technical Field

[0001] The present application relates to the technical field of text recognition, and in particular to a text recognition method, device, electronic device, and computer-readable storage medium for complex scenes. Background Art

[0002] Tibetan characters in natural scenes contain rich information and have great research and practical value. Due to the complex background, diverse layout, overlapping text, multi-direction, multi-scale, and diverse font styles of natural scene images, Tibetan text recognition under natural scene conditions faces many challenges. The existing methods for Tibetan text line recognition include: end-to-end text recognition methods. The inventors found through research that the existing end-to-end recognition method unifies detection and recognition into the same model to share image features, and the two tasks can interact with each other. With the continuous change of the architecture from the convolutional neural network architecture to the query-based network architecture, the detection and recognition effects are constantly improving, but the text recognition method in the prior art still needs to be improved. Summary of the invention

[0003] The embodiments of the present application provide a text recognition method and device, which are helpful to improve the accuracy of text recognition.

[0004] In a first aspect, an embodiment of the present application provides a text recognition method, comprising:

[0005] Through the feature extraction module of the pre-trained text recognition model, deep multi-scale feature extraction is performed on the text image to obtain multi-scale image features;

[0006] The multi-scale image features are encoded by an encoder of the text recognition model to obtain encoded features;

[0007] Performing text line area selection processing on the encoding feature through the area selection module of the text recognition model to obtain a first query vector corresponding to the text line;

[0008] Performing layer-by-layer decoding processing based on the first query vector and the encoding feature through a decoder of the text recognition model to obtain a second query vector, wherein the layer-by-layer decoding processing is performed by fusing multimodal features;

[0009] Predicting text-related information on the second query vector using a plurality of different first prediction heads of the text recognition model to obtain text information prediction results corresponding to each of the first prediction heads;

[0010] According to the text information prediction result, a text recognition result of the text image is obtained.

[0011] In a second aspect, an embodiment of the present application provides a text recognition device, including:

[0012] A multi-scale image feature acquisition module is used to perform deep multi-scale feature extraction on text images through a feature extraction module of a pre-trained text recognition model to obtain multi-scale image features;

[0013] A coding feature acquisition module, used for encoding the multi-scale image features through the encoder of the text recognition model to obtain coding features;

[0014] A first query vector acquisition module, configured to perform text line area selection processing on the encoding feature through the area selection module of the text recognition model to obtain a first query vector corresponding to the text line;

[0015] A second query vector acquisition module, configured to perform layer-by-layer decoding processing based on the first query vector and the encoding feature through a decoder of the text recognition model to obtain a second query vector, wherein the layer-by-layer decoding processing is performed by fusing multimodal features;

[0016] A text information prediction module, configured to predict text-related information on the second query vector using a plurality of different first prediction heads of the text recognition model, and obtain text information prediction results corresponding to each of the first prediction heads;

[0017] The text information prediction module is further used to obtain a text recognition result of the text image according to the text information prediction result.

[0018] In a third aspect, an embodiment of the present application further discloses an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the text recognition method described in the embodiment of the present application when executing the computer program.

[0019] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of the text recognition method disclosed in the embodiment of the present application are implemented.

[0020] The text recognition method disclosed in the embodiment of the present application performs deep multi-scale feature extraction on the text image through the feature extraction module of the pre-trained text recognition model to obtain multi-scale image features; the multi-scale image features are encoded through the encoder of the text recognition model to obtain encoded features; the encoded features are subjected to text line area selection processing through the area selection module of the text recognition model to obtain the first query vector corresponding to the text line; the decoder of the text recognition model performs layer-by-layer decoding processing based on the first query vector and the encoded features to obtain the second query vector; the second query vector is predicted for text-related information through multiple different first prediction heads of the text recognition model to obtain the text information prediction results corresponding to each of the first prediction heads. This method effectively improves the accuracy of text recognition by combining multimodal features such as vision and semantics for decoding.

[0021] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0023] Figure 1 It is one of the flow charts of the text recognition method disclosed in the embodiment of the present application;

[0024] Figure 2 It is one of the structural diagrams of the text recognition model disclosed in the embodiment of the present application;

[0025] Figure 3 It is a structural diagram of a multimodal fusion module in a text recognition model disclosed in an embodiment of the present application;

[0026] Figure 4 This is the second structural diagram of the text recognition model disclosed in the embodiment of the present application;

[0027] Figure 5 This is the second flowchart of the text recognition method disclosed in the embodiment of this application;

[0028] Figure 6It is one of the structural schematic diagrams of the text recognition device disclosed in the embodiment of the present application;

[0029] Figure 7 This is the second structural diagram of the text recognition device disclosed in the embodiment of the present application;

[0030] Figure 8 A block diagram schematically shows an electronic device for executing the method according to the present application; and

[0031] Fig. 9 A storage unit for holding or carrying a program code for implementing the method according to the present application is schematically shown. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0033] Reference Figure 1 , a text recognition method disclosed in an embodiment of the present application includes: steps 110 to 160.

[0034] In the embodiment of the present application, text recognition is performed on a text image through a text recognition model to obtain a text recognition result.

[0035] like Figure 2 As shown, the text recognition model includes: a feature extraction module 210, an encoder 220, a region selection module 230, a decoder 240 and at least one first prediction head 250. The first prediction head is connected to the output of the decoder 240 respectively.

[0036] Among them, the specific implementation of the feature extraction module 210, the encoder 220 and the area selection module 230 can be found in the prior art and will not be described in detail in the embodiments of the present application.

[0037] The training method of the text recognition model is shown in the following embodiment and will not be described in detail here.

[0038] Step 110, performing deep multi-scale feature extraction on the text image through the feature extraction module of the pre-trained text recognition model to obtain multi-scale image features, performing deep multi-scale feature extraction on the text image to obtain multi-scale image features.

[0039] In some optional embodiments, the feature extraction module 210 may adopt a ResNet network structure. That is, the text recognition model uses ResNet as the backbone network to extract multi-scale features. Taking the image size of the input text recognition model as H×W as an example, after the feature extraction module 210 extracts features from the input image, feature maps with sizes of 1 / 8, 1 / 16, and 1 / 32 of the original input image can be obtained. The feature map is then flattened, and position information is added to the feature map using position encoding to obtain image features with a size of N1×d1, where N1 is d1 is a preset feature dimension.

[0040] In some other optional embodiments, other network structures may be used to implement the feature extraction module 210 for extracting deep multi-scale features of the image input to the text recognition model, which are not listed one by one in the embodiments of the present application. The specific network structure for implementing the feature extraction module 210 is not limited in the embodiments of the present application.

[0041] Step 120: encoding the multi-scale image features through the encoder of the text recognition model to obtain encoded features.

[0042] In some optional embodiments, the encoder may adopt an encoder based on a deformable attention mechanism. The multi-scale image features output by the feature extraction module 210 are used as encoder input, and the multi-scale image features are first further encoded by the encoder to obtain encoded features. Taking the multi-scale image feature size of the input encoder as N1×d1 as an example, the encoded features can be represented as a feature map of N1×d2, where d2 represents the dimension of the encoding vector in the feature map, and each d2-dimensional encoding vector can represent the features of a specific position in the text image input to the text recognition model.

[0043] Step 130 , performing text line area selection processing on the encoding feature through the area selection module of the text recognition model to obtain a first query vector corresponding to the text line.

[0044] Optionally, the region selection module of the text recognition model is used to perform text line region selection processing on the encoded features to obtain a first query vector corresponding to the text line, including: performing feature mapping processing on the encoded features through the region selection module of the text recognition model to obtain a Bezier center curve for representing the position of the text line in the text image, and obtaining the first query vector based on the Bezier center curve.

[0045] In some optional embodiments, the region selection module 230 may include two groups of fully connected layers. After the feature map output by the encoder is input to the region selection module 230, each pixel in the feature map passes through a group of fully connected layers, and the last fully connected layer in the group of fully connected layers outputs the offset of a specified number of Bezier control points. The offset of the Bezier control point is used to determine the Bezier curve. Taking the encoding vector dimension of the last fully connected layer of the region selection module 230 as 8 as an example, the region selection module 230 can output the offset of 4 Bezier control points. Assuming p i =(p ix ,p iy )∈[0,1] 2 represents a pixel point on the feature map input to the region selection module 230, and its corresponding four Bezier control points are in σ is the sigmoid function (i.e., activation function). Represents the offset of the Bezier control point, which is predicted by the fully connected layer. The central Bezier curve can be determined based on the Bezier control point. In some optional embodiments, the central Bezier curve is used to express the position of the text line, which can not only effectively fit the scene text without overlapping problems, but also achieve the effect of simple annotation. The Bezier curve can accurately fit the position of the curved text line, which helps to improve the accuracy of text line detection.

[0046] On the other hand, the region selection module 230 uses another set of fully connected layers to output the confidence that the region corresponding to the central Bezier curve is a text line, thereby selecting the first K central Bezier curves with the highest confidence, where K is an integer greater than 1.

[0047] Furthermore, given the first K central Bezier curves corresponding to the first query vector, N2 points are uniformly sampled on each central Bezier curve. The points sampled from the central Bezier curve can cover the characters in the text, and each text image can obtain K×N2×2 coordinate information. After position embedding and full connection layer, the first query vector P of K×N2×256 is obtained. q .

[0048] Step 140: Perform layer-by-layer decoding processing based on the first query vector and the encoding feature through the decoder of the text recognition model to obtain a second query vector.

[0049] The layer-by-layer decoding process is performed by fusing multi-modal features.

[0050] Next, the decoded features output by the encoder and the first query vector output by the region selection module 230 are used as inputs of the decoder for decoding processing.

[0051] In some optional embodiments, a decoder of the text recognition model performs layer-by-layer decoding processing based on the first query vector and the encoding feature to obtain a second query vector, including: fusing an initialized learnable query vector and the first query vector to obtain a fused query vector; and a decoder of the text recognition model performs layer-by-layer decoding processing based on the fused query vector and the encoding feature to obtain a second query vector.

[0052] The input of the decoder 240 includes two parts of input, one is the output of the encoder, and the other is the fusion query F q In addition, the initialization can learn the query L q , the first query vector P obtained in the previous step q and learnable query L q Add together to get the final fusion query vector F q , F q =P q +L q Then, the output encoding features of encoder 230 are combined with the fused query vector F q They are sent to the decoder 230 for decoding to obtain a decoding result.

[0053] In some optional embodiments, such as Figure 2 As shown, the decoder 240 further includes: a first decoding layer 2401 and at least one second decoding layer 2402. Among them, the first decoding layer is a first type decoding layer, and the other decoding layers are second type decoding layers. The decoder 240 further includes the following decoding layers: a first decoding layer 2401 and at least one second type decoding layer in cascade, and the second decoding layer includes: a first type decoding layer and a multimodal fusion module. For example, the first decoding layer 2401 can be composed of two multi-head attention modules, a variability attention module and a feedforward network, and correspondingly, the second decoding layer 2402 can be composed of a multimodal fusion module and two multi-head attention modules, a variability attention module and a feedforward network. The first prediction head 250 is connected to the output of each decoding layer respectively.

[0054] In some optional embodiments, the first prediction head includes: a Bezier center curve prediction head corresponding to each decoding layer, the decoder includes the following decoding layers: a first type of decoding layer and at least one second type of decoding layer in cascade, the second type of decoding layer includes: a multimodal fusion module and a first type of decoding layer, the decoder of the text recognition model performs layer-by-layer decoding processing based on the fused query vector and the encoding feature to obtain a second query vector, including: performing decoding processing based on the fused query vector and the encoding feature through the first layer of the first type of decoding layer to obtain a query vector output by the first layer of the first type of decoding layer; using the first layer of the first type of decoding layer as the previous first type of decoding layer, and using the first layer of the second type of decoding layer as the current second type of decoding layer, performing the following decoding processing to obtain a second query vector; using the Bezier center curve prediction head based on the query vector output by the previous first type of decoding layer The vector performs text line position regression prediction to obtain a first query vector updated by the previous decoding process; the query vector output by the previous first-class decoding layer is extracted and fused by the multimodal fusion module of the current second-class decoding layer to obtain a fused vector; the first-class decoding layer in the current second-class decoding layer performs decoding based on the fused vector, the first query vector updated by the previous decoding process and the encoding feature to obtain the query vector output by the current second-class decoding layer; when the current second-class decoding layer is the last second-class decoding layer, the query vector output by the current second-class decoding layer is used as the second query vector; otherwise, the first-class decoding layer in the current second-class decoding layer is used as the previous first-class decoding layer, and the second-class decoding layer next to the current second-class decoding layer is used as the current second-class decoding layer, and the decoding process of the next layer is iteratively performed.

[0055] The first prediction head connected to each decoding layer shares parameters.

[0056] In the process of layer-by-layer decoding by the decoder, the input of the first type of decoding layer is the fused query vector F qand the encoding features output by the encoder, the input of the second type of decoding layer is the encoding features, the query vector output by the previous layer of decoding layer and the first query vector updated by regression prediction of the Bezier center curve prediction head. For example, the input of the multimodal fusion module in the second layer of decoding layer (i.e., the first layer of the second type of decoding layer) is the query vector output by the first layer of decoding layer (i.e., the first layer of the first type of decoding layer); the input of the multimodal fusion module in the third layer of decoding layer (i.e., the second layer of the second type of decoding layer) is the query vector output by the first type of decoding layer in the first layer of the second type of decoding layer (i.e., the first layer of the second type of decoding layer). The input of the multimodal fusion module in the second type of decoding layer is the query vector output by the previous layer. The multimodal fusion module fuses the query vector with the corresponding semantic information, and then adds it to the first query vector updated in the previous layer for the subsequent decoding process.

[0057] The specific implementation of the first decoding layer 2401 refers to the decoding network structure in the prior art, which will not be described in detail here. The first decoding layer 2401 is used to decode the fused query vector F input to the first decoding layer 2401. q The encoded features output by the encoder are decoded and the query vector is output.

[0058] Optionally, the query vector output by the previous first-category decoding layer is subjected to semantic information extraction and fusion processing by the multimodal fusion module of the current second-category decoding layer to obtain a fusion vector, including: performing text classification prediction on the query vector output by the previous first-category decoding layer to obtain a semantic information encoding vector corresponding to the text prediction result; using the query vector output by the previous first-category decoding layer as the query input and the semantic information encoding vector as the key and value input, extracting semantic information based on a masked attention mechanism to obtain a semantic feature vector; performing weighted fusion on the semantic feature vector and the query vector output by the previous first-category decoding layer to obtain a multimodal fusion vector; performing feature mapping processing on the multimodal fusion vector to obtain a fusion vector output by the multimodal fusion module of the current layer.

[0059] In some optional embodiments, such as Figure 3 As shown, the multimodal fusion module includes: a masked attention mechanism layer 310, a character recognition layer 320, a feature fusion layer 330 and a fully connected layer 340. The structure of the masked attention mechanism layer 310 can be found in the prior art, and the specific implementation of the character recognition layer 320 can be found in the prediction head of the character recognition model in the technology.

[0060] In the specific implementation, the query vector output by the previous first-category decoding layer is firstly predicted by the character recognition layer 320 to obtain the text prediction result, and then the text prediction result is encoded by the word vector encoding to obtain the semantic information encoding vector to capture the text semantic information in the query vector. Then, the query vector output by the previous first-category decoding layer is used as the query input Query of the masked attention mechanism layer 310, and the semantic information encoding vector is used as the key and value input Key and Value of the masked attention mechanism layer 310. The semantic features are extracted by the masked attention mechanism layer 310 to obtain the semantic feature vector corresponding to the query vector. The role of the mask is to prevent a certain dimension feature of the query vector from being calculated similarly to itself. By fusing different types of information, the features are enriched, thereby improving the accuracy of text recognition.

[0061] Furthermore, the semantic feature vector and the query vector output by the previous first-class decoding layer are weightedly fused by the feature fusion layer 330 to obtain a multimodal fusion vector, which is represented by the symbol F text , the query vector F is input into the multimodal fusion module query , the feature F obtained by splicing on the feature dimension concat Send to the fully connected layer 340 to predict a′ f , a′ f Indicates F concat The weight of each dimension feature, and then the weight a′ f Divide into two equal parts into the first weight a′ f1 and the second weight a′ f2 , respectively with the concatenation vector F text , query vector F query After multiplication, add them together to get the fusion vector.

[0062] By performing layer-by-layer decoding processing through the decoder described above, a feature vector of size K×N2×256, i.e., the second query vector, can be obtained, wherein K represents the maximum number of text lines that the text recognition model can detect, and N2×256 represents the feature vector size of each text line.

[0063] Step 150: Predict text-related information on the second query vector using a plurality of different first prediction heads of the text recognition model to obtain text information prediction results corresponding to each of the first prediction heads.

[0064] The decoder decodes and outputs a second query vector with a shape of K×N2×256, and the second query vector is sent to different first prediction heads for prediction of a specific task.

[0065] In an embodiment of the present application, the first prediction head includes but is not limited to: a text line classification prediction head, a character classification prediction head, a Bezier center curve prediction head, and a Bezier boundary curve prediction head. Accordingly, the text information prediction results corresponding to the first prediction heads include one or more of the following: text line classification prediction results, character classification prediction results, Bezier center curve prediction results, and Bezier boundary curve prediction results. Optionally, the text line classification prediction head is composed of a fully connected layer for determining whether it is a text instance; the character classification prediction head is composed of a fully connected layer for implementing character classification in each text example; the Bezier center curve prediction head is composed of multiple fully connected layers for regressing the Bezier center curve representing the position of the text line; the Bezier boundary curve prediction head is composed of multiple fully connected layers for regressing the Bezier boundary curve of the text line.

[0066] In some optional embodiments, such as Figure 4 As shown, the text recognition model further includes: a second prediction head 260, which is composed of a mask branch network. The second prediction head 260 is used to predict the mask map of the text line. The text information prediction result corresponding to the second prediction head 260 is used to indicate the position of the text line.

[0067] Accordingly, refer to Figure 5 , after the decoder of the text recognition model performs layer-by-layer decoding processing based on the first query vector and the encoding feature to obtain the second query vector, the method further includes: step 1402 and step 1404.

[0068] Step 1402, fusing the multi-scale image features and the encoding features through the second prediction head of the text recognition model to obtain a mask fusion feature vector;

[0069] Step 1404: Perform mask information prediction based on the mask fusion feature vector and the second query vector through the second prediction head to obtain a text information prediction result corresponding to the second prediction head.

[0070] Optionally, the multi-scale image features and the coding features are fused through the second prediction head of the text recognition model to obtain a mask fusion feature vector, including: obtaining the image features of the first scale and the coding features of the second scale in the multi-scale image features through the second prediction head of the text recognition model; and performing feature fusion on the image features of the first scale and the coding features of the second scale through the second prediction head to obtain a mask fusion feature vector.

[0071] The second prediction head obtains the mask prediction result according to the mask fusion feature vector and the second query vector. For example, the 1 / 4 image feature is obtained from the multi-scale image feature extracted by the feature extraction module 210 to obtain the multi-scale image feature F extract On the other hand, 1 / 8 of the encoding features are obtained from the encoding features output by the encoder to obtain the encoding features F encoder . Further, the multi-scale image feature F extract After the convolution layer, the feature dimension is adjusted to obtain F lateral , and then the encoding feature F encoder Perform 2x upsampling with F lateral Add them together to get F mask , and finally pass through a convolutional layer to F mask By performing feature extraction, the mask fusion feature vector can be obtained.

[0072] Furthermore, the second query vector is passed through multiple fully connected layers to obtain a mask embedding vector, and then the mask embedding vector and the mask fusion feature vector are matrix multiplied to obtain a mask prediction result, that is, a text information prediction result corresponding to the second prediction head.

[0073] Step 160: Obtain a text recognition result of the text image according to the text information prediction result.

[0074] As described above, in some optional embodiments, the text recognition model includes: a text line classification prediction head, a character classification prediction head, a Bezier center curve prediction head, and a Bezier boundary curve prediction head. Accordingly, according to the text information prediction result, the text recognition result of the text image is obtained, including: according to the text line classification prediction result, the character classification prediction result, the Bezier center curve prediction result, and the Bezier boundary curve prediction result, the text recognition result of the text line in the text image is obtained. Among them, the text line classification prediction result is used to indicate: whether the instance obtained by recognition is a text line; the character classification prediction result is used to indicate the character recognition result corresponding to the text line; the Bezier center curve prediction result is used to indicate the center position of the text line; the Bezier boundary curve prediction result is used to indicate the boundary position of the text line.

[0075] In some optional embodiments, the text recognition model includes: a text line classification prediction head, a character classification prediction head, a Bezier center curve prediction head, a Bezier boundary curve prediction head, and a second prediction head. Accordingly, according to the text information prediction result, the text recognition result of the text image is obtained, including: according to the text line classification prediction result, the character classification prediction result, the Bezier center curve prediction result, the Bezier boundary curve prediction result, and the mask prediction result of the text line, the text recognition result of the text line in the text image is obtained. Wherein, the mask prediction result of the text line is used to indicate the position of the text line.

[0076] After obtaining the text line classification prediction result, the character classification prediction result, the Bezier center curve prediction result, and the Bezier boundary curve prediction result, the position information and character recognition result of each text line contained in the text image can be obtained.

[0077] In order to make this solution clearer, the training process of the text recognition model is described below by taking the text recognition model including: feature extraction module, encoder, region selection module, decoder, second prediction head, text line classification prediction head, character classification prediction head, Bezier center curve prediction head, Bezier boundary curve prediction head and second prediction head as an example. The second prediction head is composed of a mask branch network.

[0078] In the process of training the text recognition model, the loss function of the text recognition model is configured as the sum of the encoding loss of the encoder and the decoding loss of the decoder, wherein the decoding loss of the decoder includes: text line classification prediction loss, character classification prediction loss, Bezier center curve prediction loss, Bezier boundary curve prediction loss and text line mask prediction loss.

[0079] The text recognition model is trained by the following method: through the feature extraction module of the text recognition model, deep multi-scale feature extraction is performed on the sample image to obtain multi-scale image features; through the encoder, the multi-scale image features are encoded to obtain encoding features; through the region selection module, the encoding features are subjected to text line region selection processing to obtain the first query vector corresponding to the text line; through the decoder, layer-by-layer decoding processing is performed based on the first query vector and the encoding features to obtain the second query vector; through the first prediction head, the second query vector is predicted for text-related information to obtain the text information prediction results corresponding to each of the first prediction heads; through the second prediction head of the text recognition model, the multi-scale image features and the encoding features are fused to obtain a mask fusion feature vector; through the second prediction head, mask information is predicted based on the mask fusion feature vector and the second query vector to obtain the text information prediction result corresponding to the second prediction head; the model loss is calculated according to the sum of the encoding loss corresponding to the encoder and the decoding loss of the decoder; by optimizing the model loss, the text recognition model is iteratively optimized. Wherein, the encoding loss corresponding to the encoder includes:

[0080] The following example illustrates how to calculate the model loss of a text recognition model.

[0081] Referring to the query-based target detection method in the prior art, the Hungarian algorithm is first used to calculate the label set G and the prediction set The optimal allocation of Minimize the matching loss. The label set is the label set of the sample images in the training sample; the prediction set is the text line position prediction result and recognition result obtained after performing the above multi-level feature extraction, feature encoding, decoding processing, and text line prediction operations on the sample images in the training sample. The optimal allocation can be calculated by the following formula:

[0082]

[0083] Where M is the number of text line instances in the sample image, L match Represents the matching loss calculation formula, which can be used to calculate the matching loss of the encoder and the matching loss of the decoder respectively.

[0084] Among them, the matching loss function of the encoder is as follows:

[0085]

[0086] The matching loss function of the decoder is as follows:

[0087]

[0088] Among them, λ cls_enc , pos_enc , cls_dec , pos_dec , char_dec is a hyperparameter, indicating the weight of each loss; L cls Indicates the use of focal loss with an increased balancing factor to calculate the text line instance classification loss; L pos Represents L1distanceloss, which is used to calculate the regression coordinate loss of the Bezier curve; L char Represents CTC loss, which is used to calculate character classification loss.

[0089] Next, based on the optimal allocation of the label set and the prediction set, the model loss is calculated. Optionally, the model loss of the text recognition model can be expressed as: L all =L enc +L dec , where L all represents the model loss, L enc represents the coding loss, L dec Represents the decoding loss. Various types of losses can be calculated by the following formula:

[0090]

[0091] Among them, L represents the number of feature points, K represents the number of queries, l represents the lth feature point, k represents the kth query, and λ′ cls_enc ,λ′ cpos_enc ,λ′ cls_dec ,λ′ cp0s_dec ,λ′ tbpos_dec ,λ′ char_dec ,λ′ char_recg and λ′ mask_dec is a hyperparameter.

[0092] Used below replace represents the text classification loss,

[0093] use Unified replacement and The specific calculation method is as follows:

[0094] 1) Represents text classification loss

[0095]

[0096] Among them, 1 represents the indicator function, Im(μ) represents the image of mapping μ, Indicates that the prediction target of the *th feature point or query is the text instance probability, that is, the coordinate prediction loss of the Bezier center curve.

[0097] 2) Bezier center curve prediction loss

[0098]

[0099] Among them, cNum represents the number of sampling points on the central Bezier curve. Represents the coordinates of the nth sampling point on the central Bezier curve of the *th feature point or the corresponding sample label of the query. Represents the coordinates of the nth sampling point on the central Bezier curve of the *th feature point or query prediction.

[0100] 3) Bezier boundary curve prediction loss

[0101]

[0102] Among them, tpNum represents the number of sampling points on the boundary Bezier curve. Indicates the coordinates of the nth sampling point on the upper curve of the sample label corresponding to the kth query, represents the coordinates of the nth sampling point on the upper curve of the kth query prediction, Indicates the coordinates of the nth sampling point of the lower curve corresponding to the kth query label, Represents the coordinates of the nth sampling point on the lower curve of the kth query prediction.

[0103] 4) Character classification prediction loss

[0104]

[0105] in, represents the sample label text corresponding to the k-th query, represents the predicted text of the k-th query and CTC() represents the transcription function.

[0106] 5) Mask prediction loss

[0107]

[0108] in, represents the label mask corresponding to the k-th query, represents the predicted mask of the k-th query, CE() represents CE loss, and DICE() represents DICE loss.

[0109] In some optional embodiments, when the text recognition model includes a second prediction head, the mask prediction loss is included when calculating the model loss.

[0110] Among them, the specific implementation methods of the steps performed on the sample images when training the text recognition model can refer to the steps of data processing of the inferred sample images through the text recognition model in the previous embodiments, and will not be repeated in the embodiments of this application.

[0111] In some other optional embodiments, when the text recognition model does not include the second prediction head, the mask prediction loss is not included when calculating the model loss.

[0112] In an embodiment of the present application, text line detection is regarded as a set prediction problem, and the sample labels of the text lines and the prediction results are matched one-to-one. Each sample label of each text line has a query corresponding to it. The number of queries is fixed and will not generate a large number of duplicate targets. A certain degree of target overlap problem can be solved without the need for post-processing.

[0113] In summary, the text recognition method disclosed in the embodiment of the present application performs deep multi-scale feature extraction on the text image through the feature extraction module of the pre-trained text recognition model to obtain multi-scale image features; the multi-scale image features are encoded through the encoder of the text recognition model to obtain encoded features; the encoded features are subjected to text line area selection processing through the area selection module of the text recognition model to obtain the first query vector corresponding to the text line; the decoder of the text recognition model performs layer-by-layer decoding processing based on the first query vector and the encoded features to obtain the second query vector; the second query vector is predicted for text-related information through multiple different first prediction heads of the text recognition model to obtain the text information prediction results corresponding to each of the first prediction heads. This method effectively improves the accuracy of text recognition by combining multimodal features such as vision and semantics for decoding.

[0114] On the other hand, by configuring a second prediction head for mask prediction in the text recognition model, the effect of text line detection can be improved. By setting the second prediction head and the first prediction head for text line classification prediction, character classification prediction, Bezier center curve prediction and Bezier boundary curve prediction, and combining multiple task training text recognition models, the text line recognition accuracy of the text recognition model can be further improved.

[0115] Reference Figure 6 The present application also discloses a text recognition device, which includes:

[0116] A multi-scale image feature acquisition module 610 is used to perform deep multi-scale feature extraction on the text image through a feature extraction module of a pre-trained text recognition model to obtain multi-scale image features;

[0117] A coding feature acquisition module 620 is used to encode the multi-scale image features through the encoder of the text recognition model to obtain coding features;

[0118] A first query vector acquisition module 630 is used to perform text line area selection processing on the encoding feature through the area selection module of the text recognition model to obtain a first query vector corresponding to the text line;

[0119] A second query vector acquisition module 640 is configured to perform layer-by-layer decoding processing based on the first query vector and the encoding feature through a decoder of the text recognition model to obtain a second query vector, wherein the layer-by-layer decoding processing is performed by fusing multimodal features;

[0120] A text information prediction module 650, configured to predict text-related information on the second query vector using a plurality of different first prediction heads of the text recognition model, and obtain text information prediction results corresponding to each of the first prediction heads;

[0121] The text information prediction module 650 is further configured to obtain a text recognition result of the text image according to the text information prediction result.

[0122] Optionally, the second query vector obtaining module 640 is further configured to:

[0123] Fusing the initialized learnable query vector and the first query vector to obtain a fused query vector;

[0124] A decoder of the text recognition model performs layer-by-layer decoding processing based on the fused query vector and the encoded features to obtain a second query vector.

[0125] Optionally, the first prediction head includes: a Bezier center curve prediction head, the decoder includes the following decoding layers: a first type of decoding layer and at least one second type of decoding layer in cascade, the second type of decoding layer includes: a multimodal fusion module and a first type of decoding layer, the decoder of the text recognition model performs layer-by-layer decoding processing based on the fused query vector and the encoding feature to obtain a second query vector, including:

[0126] By means of the first-class decoding layer of the first layer, decoding processing is performed based on the fused query vector and the encoding feature to obtain a query vector output by the first-class decoding layer of the first layer;

[0127] The first-class decoding layer of the first layer is used as the previous first-class decoding layer, and the second-class decoding layer of the first layer is used as the current second-class decoding layer, and the following decoding processing is performed to obtain a second query vector;

[0128] Performing text line position regression prediction based on the query vector output by the previous first-class decoding layer through the Bezier center curve prediction head to obtain a first query vector updated by the previous layer decoding process;

[0129] Performing semantic information extraction and fusion processing on the query vector output by the previous first-class decoding layer through the multimodal fusion module of the current second-class decoding layer to obtain a fusion vector;

[0130] By performing decoding processing on the first type of decoding layer in the current second type of decoding layer based on the fusion vector, the first query vector updated after the previous layer decoding processing, and the encoding feature, a query vector output by the current second type of decoding layer is obtained;

[0131] In the case that the current second-category decoding layer is the last second-category decoding layer, the query vector output by the current second-category decoding layer is used as the second query vector; otherwise, the first-category decoding layer in the current second-category decoding layer is used as the previous first-category decoding layer, and the second-category decoding layer below the current second-category decoding layer is used as the current second-category decoding layer, and the next layer of decoding processing is iteratively performed.

[0132] Optionally, performing semantic information extraction and fusion processing on the query vector output by the previous first-category decoding layer through the multimodal fusion module of the current second-category decoding layer to obtain a fusion vector includes:

[0133] Performing text classification prediction on the query vector output by the previous first-category decoding layer to obtain a semantic information encoding vector corresponding to the text prediction result;

[0134] Using the query vector output by the previous first-category decoding layer as the query input and the semantic information encoding vector as the key and value input, extracting the semantic information based on the masked attention mechanism to obtain a semantic feature vector;

[0135] Performing weighted fusion on the semantic feature vector and the query vector output by the previous first-category decoding layer to obtain a multimodal fusion vector;

[0136] Perform feature mapping processing on the multimodal fusion vector to obtain a fusion vector output by the multimodal fusion module of the current layer.

[0137] Optional, such as Figure 7 As shown, the device also includes:

[0138] The mask prediction module 660 is used to fuse the multi-scale image features and the encoding features through the second prediction head of the text recognition model to obtain a mask fusion feature vector; and to predict mask information based on the mask fusion feature vector and the second query vector through the second prediction head to obtain a text information prediction result corresponding to the second prediction head.

[0139] Optionally, in the process of training the text recognition model, the loss function of the text recognition model is configured as the sum of the encoding loss of the encoder and the decoding loss of the decoder, wherein the decoding loss of the decoder includes: text line classification prediction loss, character classification prediction loss, Bezier center curve prediction loss, Bezier boundary curve prediction loss and text line mask prediction loss.

[0140] The text recognition device disclosed in the embodiment of the present application is used to implement the text recognition method described in the embodiment of the present application. The specific implementation methods of each module of the device will not be repeated here. Please refer to the specific implementation methods of the corresponding steps in the method embodiment.

[0141] The text recognition device disclosed in the embodiment of the present application performs deep multi-scale feature extraction on the text image through the feature extraction module of the pre-trained text recognition model to obtain multi-scale image features; the multi-scale image features are encoded through the encoder of the text recognition model to obtain encoded features; the encoded features are subjected to text line area selection processing through the area selection module of the text recognition model to obtain the first query vector corresponding to the text line; the decoder of the text recognition model performs layer-by-layer decoding processing based on the first query vector and the encoded features to obtain the second query vector; the second query vector is predicted for text-related information through multiple different first prediction heads of the text recognition model to obtain the text information prediction results corresponding to each of the first prediction heads. This device effectively improves the accuracy of text recognition by combining multimodal features such as vision and semantics for decoding.

[0142] On the other hand, by configuring a second prediction head for mask prediction in the text recognition model, the effect of text line detection can be improved. By setting the second prediction head and the first prediction head for text line classification prediction, character classification prediction, Bezier center curve prediction and Bezier boundary curve prediction, and combining multiple task training text recognition models, the text line recognition accuracy of the text recognition model can be further improved.

[0143] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0144] The above is a detailed introduction to a text recognition method and device provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for a person skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

[0145] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0146] The various component embodiments of the present application can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all functions of some or all components in the electronic device according to the embodiment of the present application. The present application can also be implemented as a device or apparatus program (e.g., computer program and computer program product) for executing part or all of the methods described herein. Such a program implementing the present application can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0147] For example, Figure 8An electronic device that can implement the method according to the present application is shown. The electronic device can be a PC, a mobile terminal, a personal digital assistant, a tablet computer, etc. The electronic device traditionally includes a processor 810 and a memory 820 and a program code 830 stored on the memory 820 and can be run on the processor 810, and the processor 810 implements the method described in the above embodiment when executing the program code 830. The memory 820 can be a computer program product or a computer-readable medium. The memory 820 can be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk or a ROM. The memory 820 has a storage space 8201 for the program code 830 of the computer program for executing any method step in the above method. For example, the storage space 8201 for the program code 830 can include various computer programs for implementing the various steps in the above method respectively. The program code 830 is a computer-readable code. These computer programs can be read from one or more computer program products or written into the one or more computer program products. These computer program products include program code carriers such as hard disks, compact disks (CDs), memory cards or floppy disks. The computer program includes a computer readable code, and when the computer readable code is executed on an electronic device, the electronic device is caused to execute the method according to the above embodiment.

[0148] The embodiment of the present application also discloses a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the text recognition method described in the embodiment of the present application are implemented.

[0149] Such a computer program product may be a computer-readable storage medium, which may have a computer-readable storage medium having a Figure 8 The memory 820 in the electronic device shown in the figure is similar to the storage segment, storage space, etc. The program code can be compressed and stored in the computer-readable storage medium in an appropriate form. The computer-readable storage medium is generally as shown in the reference Fig. 9 The portable or fixed storage unit. Generally, the storage unit includes computer readable code 830', which is a code read by a processor. When the code is executed by the processor, each step in the method described above is implemented.

[0150] The term "one embodiment", "embodiment" or "one or more embodiments" herein means that a particular feature, structure or characteristic described in conjunction with the embodiment is included in at least one embodiment of the present application. In addition, please note that the examples of the term "in one embodiment" here do not necessarily all refer to the same embodiment.

[0151] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0152] In the claims, any reference signs placed between brackets shall not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present application may be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third etc. does not indicate any order. These words may be interpreted as names.

[0153] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A text recognition method, characterized in that: The method comprises: Through the feature extraction module of the pre-trained text recognition model, deep multi-scale feature extraction is performed on the text image to obtain multi-scale image features; The multi-scale image features are encoded by an encoder of the text recognition model to obtain encoded features; Performing text line area selection processing on the encoding feature through the area selection module of the text recognition model to obtain a first query vector corresponding to the text line; Performing layer-by-layer decoding processing based on the first query vector and the encoding feature through a decoder of the text recognition model to obtain a second query vector, wherein the layer-by-layer decoding processing is performed by fusing multimodal features; Predicting text-related information on the second query vector using a plurality of different first prediction heads of the text recognition model to obtain text information prediction results corresponding to each of the first prediction heads; According to the text information prediction result, a text recognition result of the text image is obtained.

2. The method according to claim 1, characterized in that The step of performing layer-by-layer decoding processing based on the first query vector and the encoding feature by the decoder of the text recognition model to obtain a second query vector includes: Fusing the initialized learnable query vector and the first query vector to obtain a fused query vector; A decoder of the text recognition model performs layer-by-layer decoding processing based on the fused query vector and the encoded features to obtain a second query vector.

3. The method according to claim 2, characterized in that The first prediction head includes: a Bezier center curve prediction head, the decoder includes the following decoding layers: a first type of decoding layer and at least one second type of decoding layer in cascade, the second type of decoding layer includes: a multimodal fusion module and a first type of decoding layer, the decoder of the text recognition model performs layer-by-layer decoding processing based on the fused query vector and the encoding feature to obtain a second query vector, including: By means of the first-class decoding layer of the first layer, decoding processing is performed based on the fused query vector and the encoding feature to obtain a query vector output by the first-class decoding layer of the first layer; The first-class decoding layer of the first layer is used as the previous first-class decoding layer, and the second-class decoding layer of the first layer is used as the current second-class decoding layer, and the following decoding processing is performed to obtain a second query vector; Performing text line position regression prediction based on the query vector output by the previous first-class decoding layer through the Bezier center curve prediction head to obtain a first query vector updated by the previous layer decoding process; Performing semantic information extraction and fusion processing on the query vector output by the previous first-class decoding layer through the multimodal fusion module of the current second-class decoding layer to obtain a fusion vector; By performing decoding processing on the first type of decoding layer in the current second type of decoding layer based on the fusion vector, the first query vector updated after the previous layer decoding processing, and the encoding feature, a query vector output by the current second type of decoding layer is obtained; In the case that the current second-category decoding layer is the last second-category decoding layer, the query vector output by the current second-category decoding layer is used as the second query vector; otherwise, the first-category decoding layer in the current second-category decoding layer is used as the previous first-category decoding layer, and the second-category decoding layer below the current second-category decoding layer is used as the current second-category decoding layer, and the next layer of decoding processing is iteratively performed.

4. The method according to claim 3, characterized in that The multimodal fusion module of the current second-class decoding layer performs semantic information extraction and fusion processing on the query vector output by the previous first-class decoding layer to obtain a fusion vector, including: Performing text classification prediction on the query vector output by the previous first-category decoding layer to obtain a semantic information encoding vector corresponding to the text prediction result; Using the query vector output by the previous first-category decoding layer as the query input and the semantic information encoding vector as the key and value input, extracting the semantic information based on the masked attention mechanism to obtain a semantic feature vector; Performing weighted fusion on the semantic feature vector and the query vector output by the previous first-category decoding layer to obtain a multimodal fusion vector; Perform feature mapping processing on the multimodal fusion vector to obtain a fusion vector output by the multimodal fusion module of the current layer.

5. The method according to any one of claims 1 to 4, characterized in that: After the decoder of the text recognition model performs layer-by-layer decoding processing based on the first query vector and the encoding feature to obtain the second query vector, the method further includes: The multi-scale image features and the encoding features are fused by a second prediction head of the text recognition model to obtain a mask fusion feature vector; The second prediction head performs mask information prediction based on the mask fusion feature vector and the second query vector to obtain a text information prediction result corresponding to the second prediction head.

6. The method according to claim 5, characterized in that In the process of training the text recognition model, the loss function of the text recognition model is configured as the sum of the encoding loss of the encoder and the decoding loss of the decoder, wherein the decoding loss of the decoder includes: text line classification prediction loss, character classification prediction loss, Bezier center curve prediction loss, Bezier boundary curve prediction loss and text line mask prediction loss.

7. A text recognition device, characterized in that: The device comprises: A multi-scale image feature acquisition module is used to perform deep multi-scale feature extraction on text images through a feature extraction module of a pre-trained text recognition model to obtain multi-scale image features; A coding feature acquisition module, used for encoding the multi-scale image features through the encoder of the text recognition model to obtain coding features; A first query vector acquisition module, configured to perform text line area selection processing on the encoding feature through the area selection module of the text recognition model to obtain a first query vector corresponding to the text line; A second query vector acquisition module, configured to perform layer-by-layer decoding processing based on the first query vector and the encoding feature through a decoder of the text recognition model to obtain a second query vector, wherein the layer-by-layer decoding processing is performed by fusing multimodal features; A text information prediction module, configured to predict text-related information on the second query vector using a plurality of different first prediction heads of the text recognition model, and obtain text information prediction results corresponding to each of the first prediction heads; The text information prediction module is further used to obtain a text recognition result of the text image according to the text information prediction result.

8. The device according to claim 7, characterized in that The device also includes: A mask prediction module, configured to fuse the multi-scale image features and the encoding features through a second prediction head of the text recognition model to obtain a mask fusion feature vector; and The second prediction head performs mask information prediction based on the mask fusion feature vector and the second query vector to obtain a text information prediction result corresponding to the second prediction head.

9. An electronic device comprising a memory, a processor, and a program code stored in the memory and executable on the processor, wherein: When the processor executes the program code, the method according to any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium having program code stored thereon, characterized in that: When the program code is executed by a processor, the steps of the method described in any one of claims 1 to 6 are implemented.

Images