Multilingual end-to-end OCR algorithm and system

Abstract

The present invention provides a multilingual end-to-end OCR algorithm and system, which overcomes and bypasses the defects of the existing technology in sharding, and performs well in character adhesion, Chinese-English, and Chinese-numerical mixed data; through the self-distillation transformer module, the position relationship is retained, the parameters and model complexity are reduced, and the results are output in parallel, and the dependencies between nodes are cut off, which has stronger robustness for multi-language and multi-font scenarios, and the structure and performance are also optimized. The algorithm includes: obtaining a feature map of the image to be recognized; training the feature map through a relational attention module based on the self-distillation transformer module to obtain a character matrix; performing parallel attention decoding on the character matrix to obtain a prediction result; based on the prediction result, based on the vocabulary table, obtaining an OCR model that matches the language of the vocabulary table.

Description

Technical Field

[0001] The present invention relates to an OCR algorithm, and in particular to a multilingual end-to-end OCR algorithm and system. Background Art

[0002] In modern enterprise production environments and daily business activities, OCR (Optical Character Recognition): optical text recognition technology has been fully promoted and applied on a large scale, thanks to the growing demand for improving information entry efficiency and diversifying information carriers. In existing industrial application scenarios, considering compatibility and reliability, OCR applications are usually divided into two parts: text detection and text recognition. The text recognition network is also divided into a single-word classification recognition network structure and an overall recognition network structure. The existing overall recognition network usually adopts a CRNN (Convolutional Recurrent Neural Network) structure, which has the following disadvantages:

[0003] 1. The text image usually input is a feature sequence with spatial or planar information. When using the existing overall recognition network, it needs to be reshaped by the Recurrent layer, and the constructed context information will inevitably be lost, especially the spatial position information. If the 2D-Recurrent layer is used for encoding to solve the information loss problem, it will result in too many parameters and a complex structure.

[0004] 2. Using the existing overall recognition network, the LSTM (Long Short-Term Memory) and GRU (Gated Recurrent Unit) networks generally have requirements for the input segment width. However, in text information, there is often a mixture of Chinese and English, Chinese numbers, character adhesion, etc., which leads to different character widths. The adaptive width segment structure is difficult to have good robustness in these scenarios.

[0005] 3. For the OCR pipeline of positioning and recognition classification, it is usually difficult to achieve the optimal anchor box size selection, which requires a lengthy iteration and annotation process. Summary of the invention

[0006] In order to solve the above technical problems, the present invention provides a multilingual end-to-end OCR algorithm and system. Through a relational attention module based on a self-distillation transformer module, the positional relationship is retained, and the parameters and model complexity are reduced, so that CRNN has the advantages of stronger robustness for multilingual and multi-font scenarios.

[0007] The technical solution of the present invention is achieved in this way:

[0008] Multilingual end-to-end OCR algorithms, including:

[0009] Get the feature map of the image to be identified;

[0010] The feature graph is trained through a relational attention module to obtain a character matrix; wherein the relational attention module is a relational attention module based on a self-distillation transformer module;

[0011] Performing parallel attention decoding on the character matrix to obtain a prediction result;

[0012] According to the prediction result, based on a preset vocabulary and sentence table, an OCR model that matches the language of the vocabulary and sentence table is obtained.

[0013] Furthermore, obtaining a feature map of the image to be identified includes:

[0014] Extract the feature map based on a feature extraction layer, wherein the feature extraction layer is a Resnet45 network;

[0015] The self-distillation transformer module is a simplified transformer module obtained as follows:

[0016] Add a self-distillation transformer module after different network feature layers of the Resnet45 network to obtain the first feature series;

[0017] Performing a classification task on the first feature series to obtain a classification result;

[0018] According to the classification result, the transformer layer with expected performance and parameter amount in the self-distillation transformer module is selected to obtain a simplified transformer module.

[0019] Furthermore, the preset vocabulary and sentence table is a vocabulary and sentence table obtained by the following method:

[0020] Use the corpus of Chinese Wikipedia to generate corresponding text and image training data to obtain the benchmark corpus;

[0021] Generate a parallel corpus corresponding to the reference corpus using U non-Chinese Wikipedia corpora, where U ≥ 1;

[0022] Aligning the reference corpus and the parallel corpus at sentence level to generate the preset vocabulary and sentence table;

[0023] Furthermore, the step of obtaining an OCR model that matches the language of a preset vocabulary table according to the prediction result includes:

[0024] When the image to be identified includes a text image of a parallel language corpus, the text image is trained by a relational attention module and decoded by parallel attention to obtain a prediction result;

[0025] Decoding the prediction result based on a preset vocabulary and sentence table to obtain a translation result based on the reference corpus;

[0026] The translation results are transferred to U parallel language corpora for learning to obtain a single language OCR model.

[0027] Furthermore, performing parallel attention decoding on the character matrix to obtain a prediction result includes:

[0028] Perform linear transformation and softmax calculation on the character matrix to obtain an attention weight matrix;

[0029] Obtain a probability vector corresponding to each character in the character matrix based on the attention weight matrix and the feature map;

[0030] Perform linear transformation and softmax calculation on the probability vector to obtain the prediction result.

[0031] Furthermore, the attention weight matrix is ​​determined by the following equation:

[0032] α=softmax(W2tanh(W1O T ))

[0033] Wherein, α is the attention weight matrix, W1 and W2 are linear transformation matrices, and OT is the transposed result of the character matrix;

[0034] The probability vector is determined by the following equation:

[0035]

[0036] Among them, G is the probability vector, I is the feature map, and i is the i-th character.

[0037] Furthermore, the relation attention module comprises:

[0038] N bidirectional transformer layers, each of which includes M transformer nodes; N ≥ 2, M ≥ 2;

[0039] Any one of the transofrmet nodes in the N-1th transofrmet layer is connected to all the transofrmet nodes in the Nth transofrmet layer;

[0040] The transformer node is a transformer node based on a single attention mechanism.

[0041] Multilingual end-to-end OCR system, including:

[0042] Feature extraction layer, used to obtain the feature map of the image to be identified;

[0043] A relational attention module, used to train the feature map to obtain a character matrix; the relational attention module is a relational attention module based on a self-distillation transformer module;

[0044] The parallel attention decoding layer is used to perform parallel attention decoding on the character matrix to obtain a prediction result.

[0045] Furthermore, the relation attention module comprises:

[0046] N bidirectional transformer layers, each of which includes M transformer nodes; N ≥ 2, M ≥ 2;

[0047] Any one of the transofrmet nodes in the N-1th transofrmet layer is connected to all of the transofrmet nodes in the Nth transofrmet layer.

[0048] Furthermore, the feature extraction layer is a Resnet45 network; the self-distillation transformer modules are respectively connected after different network feature layers of the Resnet45 network;

[0049] It also includes a classifier, which is connected to the self-distillation transformer module and is used to perform a classification task on the first feature series output by the self-distillation transformer module.

[0050] Compared with the prior art, the multilingual end-to-end OCR algorithm of the first embodiment of the present invention has the following advantages:

[0051] Through the relational attention module based on the self-distilled transformer module, the existing overall recognition network has circumvented the problem of inaccurate recognition of different width characters, contiguous characters, and different font languages, while retaining the sequence and spatial position information in the text image and reducing the number of parameters. In addition, the reliance on the anchor box is eliminated, making the training process simpler and the training results better. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The accompanying drawings illustrate exemplary embodiments of the present invention and together with the description serve to explain the principles of the present invention. These drawings are included to provide a further understanding of the present invention and are incorporated in and constitute a part of this specification.

[0053] Figure 1 It is a flowchart of the multilingual end-to-end OCR algorithm of the present invention;

[0054] Figure 2 is a schematic diagram of the overall structure of the relational attention module of the present invention;

[0055] Figure 3 It is a schematic diagram of the internal structure of the transformer node of the present invention. DETAILED DESCRIPTION

[0056] The present invention will be further described in detail below in conjunction with the accompanying drawings and implementations. It is to be understood that the specific implementations described herein are only used to explain the relevant content, rather than to limit the present invention. It is also necessary to explain that, for ease of description, only the parts related to the present invention are shown in the accompanying drawings.

[0057] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments of the present invention can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0058] Embodiment 1

[0059] Reference Figure 1 , a multilingual end-to-end OCR algorithm based on self-distillation transformer and label-refinement, including:

[0060] Get the feature map of the image to be identified;

[0061] The feature map is trained through a relational attention module to obtain a character matrix; wherein the relational attention module is a relational attention module based on a self-distillation transformer module;

[0062] Perform parallel attention decoding on the character matrix to obtain the prediction results;

[0063] According to the prediction results, based on the preset vocabulary and sentence table, an OCR model that matches the language of the vocabulary and sentence table is obtained.

[0064] The relational attention module disclosed in the present invention is a structure improved from the BERT framework in the NLP field; the structure is as follows Figure 2 As shown, the relation attention module includes:

[0065] N bidirectional transformer layers, each transformer layer consists of M transformer nodes; N ≥ 2, M ≥ 2;

[0066] Any transofrmet node of the N-1th transofrmet layer is connected to all transofrmet nodes of the Nth transofrmet layer.

[0067] like Figure 3 , is the internal structure of each transformer node, where the input is F, that is, Q, K, and V are the same. Figure 3 It can be seen that in this embodiment, the transformer module removes the mask multi-head transformer structure in BERT and only retains one, that is, the transformer node is a transformer node based on a single attention mechanism, and when outputting, the entire character matrix is ​​output together to avoid the problem of the latter characters leaking information to the previous characters.

[0068] In this embodiment, the transformer model is trained based on label refinement technology in several iterations. In the first iteration, hard labels are used, that is, real labels that are either this or that. In the subsequent model training iterations, the labels output by the previous iteration are used, that is, soft labels, and the labels are different confidence levels for different classifications. Iterations are continued until the model accuracy cannot be improved any further. In this way, the model can learn the relationship between different labels, making the original network more robust to texts in different languages, as well as texts with different character widths such as mixed numbers and text.

[0069] As an optional solution of the above implementation, parallel attention decoding is performed on the character matrix to obtain the prediction result, including:

[0070] Perform linear transformation and softmax calculation on the character matrix to obtain the attention weight matrix;

[0071] Obtain the probability vector corresponding to each character in the character matrix based on the attention weight matrix and the feature map;

[0072] Perform linear transformation and softmax calculation on the probability vector to obtain the prediction result.

[0073] Among them, the attention weight matrix is ​​determined by the following equation:

[0074] α=softmax(W2tanh(W1O T ))

[0075] Where W1 and W2 are linear transformation matrices, O T Transpose the character matrix.

[0076] The probability vector G is determined by the following equation:

[0077]

[0078] Where I is the feature map and i is the i-th character.

[0079] As an optional solution of the above embodiment, obtaining a feature map of the image to be identified includes: extracting a feature map based on a feature extraction layer, where the feature extraction layer is a Resnet45 network;

[0080] The self-distilled transformer module is a streamlined transformer module obtained as follows:

[0081] Add a self-distillation transformer module after different network feature layers of the Resnet45 network to obtain the first feature series;

[0082] Perform classification tasks on the first feature series and obtain classification results;

[0083] According to the classification results, the transformer layer with expected performance and parameter amount is selected from the self-distillation transformer module to obtain the simplified transformer module.

[0084] This embodiment adds a distillation structure after different network feature layers, learns the classifier results of the next layer through distillation of a smaller classifier of the previous layer, and finally selects the number of layers with smaller loss in the transformer module to obtain a streamlined transformer module with good actual performance and a moderate parameter size. The size of the transformer module is optimized and the problem of the large space occupied by the transformer is solved. The number of layers and parameter size finally selected can be selected according to the actual application.

[0085] In addition, as a preferred implementation of this embodiment, it is generally chosen to add a distillation structure after the odd-numbered network feature layer of the Resnet45 network.

[0086] The above optimized streamlined transformer module can flexibly replace the Recurrent layer in most CRNN networks, such as bidirectional LSTM\GRU, etc.

[0087] As an optional solution of the above embodiment, the preset vocabulary table is a vocabulary table obtained by the following method:

[0088] Generate corresponding text and image training data using the first language corpus to obtain a benchmark corpus;

[0089] Generate a parallel corpus corresponding to the benchmark corpus using U parallel language corpora, where U ≥ 1;

[0090] Align the baseline corpus and the parallel corpus at the sentence level to generate a preset vocabulary and sentence table.

[0091] As an optional solution of the above implementation, the first language corpus is a corpus based on Chinese Wikipedia; the U parallel language corpora are U corpora based on non-Chinese Wikipedia. The parallel language corpora take the 7 major languages ​​of Wikipedia as an example. Specifically, the Chinese Wikipedia corpus is used to generate corresponding text image training data as the benchmark corpus of the multilingual model. In addition, the same 7 major language versions of Wikipedia are used to generate parallel corpora corresponding to the benchmark corpus. Since the same corpus source is used, the parallel corpus and the benchmark corpus can be aligned at the sentence level, and a shared vocabulary and common sentence list can be generated as the vocabulary and sentence list.

[0092] Based on the above implementation, when the image to be recognized includes a text image of a parallel language corpus, the text image is trained by the relational attention module and decoded by parallel attention to obtain a prediction result; based on the vocabulary sentence, the deep representation table of the prediction result can be decoded to obtain a translation result based on the benchmark corpus; and using this model, the translation result is transferred to U parallel language corpora for learning to obtain an OCR model of a single language.

[0093] Embodiment 2

[0094] A multilingual end-to-end OCR system based on self-distillation transformer and label-refinement, including:

[0095] Feature extraction layer, used to obtain the feature map of the image to be identified;

[0096] The relational attention module is used to train the feature map and obtain the character matrix. The relational attention module is a relational attention module based on the self-distillation transformer module.

[0097] The parallel attention decoding layer is used to perform parallel attention decoding on the character matrix to obtain the prediction results.

[0098] As an optional solution of the above embodiment, the relation attention module includes:

[0099] N bidirectional transformer layers, each transformer layer consists of M transformer nodes; N ≥ 2, M ≥ 2;

[0100] Any transofrmet node of the N-1th transofrmet layer is connected to all transofrmet nodes of the Nth transofrmet layer.

[0101] As an optional solution of the above embodiment, the feature extraction layer is a Resnet45 network; self-distillation transformer modules are respectively connected after different network feature layers of the Resnet45 network;

[0102] It also includes a classifier, which is connected to the self-distillation transformer module and is used to perform a classification task on the first feature series output by the self-distillation transformer module.

[0103] The principles and effects of this embodiment are consistent with those of the first embodiment, and will not be described again in this embodiment.

[0104] It should be understood by those skilled in the art that the above embodiments are only for the purpose of clearly illustrating the present invention, and are not intended to limit the scope of the present invention. For those skilled in the art, other changes or modifications may be made based on the above invention, and these changes or modifications are still within the scope of the present invention.

Claims

1. Multilingual end-to-end OCR algorithm, characterized by: include: Get the feature map of the image to be identified; Training the feature map through a relational attention module to obtain a character matrix; The relation attention module is a relation attention module based on a self-distillation transformer module; Performing parallel attention decoding on the character matrix to obtain a prediction result; According to the prediction result, based on a preset vocabulary and sentence table, an OCR model that matches the language of the vocabulary and sentence table is obtained; Wherein, the relationship attention module includes: N bidirectional transformer layers, each of which includes M transformer nodes; N ≥ 2, M ≥ 2; Any one of the transformer nodes in the N-1th transformer layer is connected to all of the transformer nodes in the Nth transformer layer; The transformer node is a transformer node based on a single attention mechanism; The performing parallel attention decoding on the character matrix to obtain a prediction result includes: Perform linear transformation and softmax calculation on the character matrix to obtain an attention weight matrix; Obtain a probability vector corresponding to each character in the character matrix based on the attention weight matrix and the feature map; Perform linear transformation and softmax calculation on the probability vector to obtain the prediction result.

2. The algorithm according to claim 1, characterized in that The step of obtaining a feature map of a picture to be identified includes: Extract the feature map based on a feature extraction layer, wherein the feature extraction layer is a Resnet45 network; The self-distillation transformer module is a simplified transformer module obtained as follows: Add a self-distillation transformer module after different network feature layers of the Resnet45 network to obtain the first feature series; Performing a classification task on the first feature series to obtain a classification result; According to the classification result, the transformer layer with expected performance and parameter amount in the self-distillation transformer module is selected to obtain a simplified transformer module.

3. The algorithm according to claim 1, characterized in that The preset vocabulary and sentence table is a vocabulary and sentence table obtained by the following method: Use the corpus of Chinese Wikipedia to generate corresponding text and image training data to obtain the benchmark corpus; Generate a parallel corpus corresponding to the reference corpus using U non-Chinese Wikipedia corpora, where U ≥ 1; The reference corpus and the parallel corpus are aligned at sentence level to generate the preset vocabulary and sentence table.

4. The algorithm as claimed in claim 3, characterized in that The step of obtaining an OCR model that matches the language of the vocabulary table based on a preset vocabulary table according to the prediction result includes: When the image to be identified includes a text image of a parallel language corpus, the text image is trained by a relational attention module and decoded by parallel attention to obtain a prediction result; Decoding the prediction result based on a preset vocabulary and sentence table to obtain a translation result based on the reference corpus; The translation results are transferred to U parallel language corpora for learning to obtain a single language OCR model.

5. The algorithm according to any one of claims 1 to 4, characterized in that: The attention weight matrix is ​​determined by the following equation: α=softmax(W2tanh(W1O T ) Where α is the attention weight matrix, W1 and W2 are linear transformation matrices, and T The result of transposing the character matrix; The probability vector is determined by the following equation: Among them, G is the probability vector, I is the feature map, and i is the i-th character. 6.Multilingual end-to-end OCR system, characterized by: include: Feature extraction layer, used to obtain the feature map of the image to be identified; A relation attention module, used for training the feature map to obtain a character matrix; The relation attention module is a relation attention module based on the self-distillation transformer module; A parallel attention decoding layer is used to perform parallel attention decoding on the character matrix to obtain a prediction result; wherein the relation attention module includes: N bidirectional transformer layers, each of which includes M transformer nodes; N ≥ 2, M ≥ 2; Any one of the transformer nodes in the N-1th transformer layer is connected to all of the transformer nodes in the Nth transformer layer; The performing parallel attention decoding on the character matrix to obtain a prediction result includes: Perform linear transformation and softmax calculation on the character matrix to obtain an attention weight matrix; Obtain a probability vector corresponding to each character in the character matrix based on the attention weight matrix and the feature map; Perform linear transformation and softmax calculation on the probability vector to obtain the prediction result.

7. The system according to claim 6, characterized in that The feature extraction layer is a Resnet45 network; the self-distillation transformer modules are respectively connected after different network feature layers of the Resnet45 network; It also includes a classifier, which is connected to the self-distillation transformer module and is used to perform a classification task on the first feature series output by the self-distillation transformer module.

Images