Method and device for recognizing Murankang image text based on multi-granularity decoding constraint

Abstract

The invention relates to a multigranularity decoding constraint-based Murantu image text recognition method and device, and belongs to the field of natural language processing. In order to solve the problem of poor recognition of similar characters in a Murank language image, the invention provides a Murank language text image recognition method based on multi-granularity decoding constraints, and the Murank language text image recognition method comprises the following steps of: providing a Murank language text image recognition method, wherein the Murank language text has a unique coding sequence and a unique character combination rule; the method mainly comprises the following four parts: constructing and preprocessing a Myankanu text image data set, and designing a Myankanu text image recognition model and Myankanu text image recognition model based on a multi-granularity decoding constraint, wherein the Myankanu text image recognition model comprises a sliding window-based Transformer, a language characteristic-driven multi-granularity feature extraction module and a multi-granularity feature fusion module; and the Myankanu text image recognition model comprises the following four parts: Myankanu text image data set. According to the four parts of function modularization, the Myankanu language image text recognition device based on the multi-granularity decoding constraint is manufactured, and the problem that Myankanu language similar characters are difficult to recognize due to poor image quality is effectively solved.

Description

Technical Field

[0001] The present invention discloses a Burmese image text recognition method and device based on multi-granularity decoding constraints, belonging to the technical field of natural language processing. Background Art

[0002] Burmese has a unique encoding order and character combination rules. In addition to the basic consonant characters, it also includes modifiers such as left vowel characters, upper vowel characters, right vowel characters, and lower tone characters. These modifiers can be nested in different positions of the basic characters to form complex character combinations;

[0003] A Burmese syllable may be composed of multiple independent Unicode-encoded characters. This combination and nesting feature makes the Burmese character recognition task more complicated than linear character recognition methods such as Chinese and English. The edge features of the characters are often ignored during the recognition process, resulting in misrecognition or loss of some characters during decoding, making the recognition results garbled. In addition, the high similarity between some Burmese characters also brings great interference to the Burmese recognition task. Similar characters often lead to character confusion in the recognition results. Therefore, the recognition model can only alleviate this phenomenon by capturing more fine-grained features.

[0004] In view of the above problems, the present invention proposes a method and device for Burmese image text recognition based on multi-granularity decoding constraints. On the basis of the original character sequence decoding, character cluster sequences and syllable sequences with language knowledge are introduced to establish decoding constraints, implicitly integrate language knowledge into the decoding process, and enhance the ability to distinguish similar combination characters. At the same time, through the multi-granularity feature extraction module driven by language characteristics, the visual features of different granularities of characters are adaptively captured and better aligned with the position information. Finally, the predictions of the three granularities are fused in the reasoning stage to obtain the final recognition result. Summary of the invention

[0005] The present invention proposes a Burmese image text recognition method and device based on multi-granularity decoding constraints, which is used to solve the problem of poor recognition of similar characters in Burmese images and the problem of difficulty in recognizing similar Burmese characters due to poor image quality. The present invention has a high recognition accuracy.

[0006] The technical solution of the present invention is: a method for Burmese image text recognition based on multi-granularity decoding constraints, and the specific steps of the method are as follows:

[0007] Step 1: Construction and preprocessing of Burmese text image dataset;

[0008] Step 2: Build a Burmese text image recognition model based on multi-granularity decoding constraints, including:

[0009] Step 2.1. Design a sliding window-based Transformer to extract Burmese text image features and obtain original visual features.

[0010] Step 2.2, design a multi-granularity feature extraction module driven by language characteristics: used to adaptively extract visual features of different granularities from the original visual features, and align these features with the character positions of the corresponding granularity to support the subsequent multi-granularity decoding process; wherein, the multi-granularity feature extraction module driven by language characteristics is used to capture visual features of different granularities at the character level, character cluster level, and syllable level respectively;

[0011] Step 2.3, design a multi-granularity feature fusion module to capture visual features of different granularities, use a confidence fusion strategy to perform decoding constraints, and obtain the final predicted text result; this process significantly enhances the model's recognition accuracy for easily confused similar characters in Burmese;

[0012] Step 3: Use the constructed Burmese text image recognition model based on multi-granularity decoding constraints to perform Burmese text image recognition.

[0013] Furthermore, the specific steps of Step 1 are:

[0014] Step 1.1, collect Burmese text data from the Internet, and then perform preprocessing steps such as deduplication, segmentation, and filtering of special characters to build a Burmese text label dataset for subsequent generation of Burmese text images; obtain Burmese text images of real scenes through the Internet;

[0015] Step 1.2, using the text editing network, replace or modify the text in the natural image without destroying the realism of the original image, so as to generate a text image; specifically, the text image is generated through the text conversion module, the background repair module and the fusion module;

[0016] The text conversion module is used to combine the semantics of the target text with the style of the text in the source image, including font, color, position and scale. The module introduces a skeleton-guided learning mechanism to guide the transfer of text style by predicting the text skeleton graph.

[0017] The background restoration module is responsible for erasing the original text and filling in appropriate textures. It adopts the U-Net structure and realizes background reconstruction through the encoder-decoder architecture.

[0018] The fusion module is used to effectively fuse the output information of the text conversion module and the background restoration module to generate the final edited text image, and divide the training set, test set and verification set according to the final image.

[0019] Furthermore, the Step 2.1 includes:

[0020] A sliding window-based Transformer is constructed as the backbone network, which is used to extract features from the input Burmese text image and convert the image into a token sequence. Specifically, the original image is firstly transformed into a token sequence through the patch segmentation module. The image is divided into blocks of size P×P, and then a linear embedding layer is used to project the original value feature into the dimension , and flattened into a two-dimensional vector ,in is the resolution of each tile, yes The number of channels, Represents the length of the image block sequence, H, W, and C are the height, width, and number of channels of the image respectively; the token sequence is then sent to the intra-window self-attention W-MSA and inter-window self-attention SW-MSA for modeling. W-MSA and SW-MSA can perform self-attention modeling on the image while maintaining the ability of local perception and global perception. The specific calculation process is as follows:

[0021] ;

[0022] ;

[0023] In the formula, and They represent the outputs of the W-MSA module and the SW-MSA module respectively; Represents the normalization layer, the self-attention-based representation learning calculation formula in the sliding window Transformer is:

[0024] ;

[0025] in, represents the query matrix; represents the bond matrix; represents the value matrix; represents the bias matrix, d1 represents the dimension of the image feature matrix, S represents the number of layers of the current network module, and the encoder feeds the Burmese image into the stacked sliding window Transformer. After multiple stages of network learning, the original visual features are finally obtained. Used for subsequent text recognition.

[0026] Furthermore, in the Step 2.2, the multi-granularity feature extraction module driven by the design language characteristics includes two parts: position information enhancement and multi-granularity visual information extraction, which are used to extract the original visual features. Features of different granularities are adaptively extracted and these features are aligned with the character positions of the corresponding granularity. At the same time, considering that the position information paid attention to at the same time is different when decoding at different granularities, a multi-branch visual information extraction module is designed to focus on the visual features of characters, character clusters and syllable granularities respectively for subsequent multi-granularity decoding.

[0027] Furthermore, the Step 2.2 includes:

[0028] Step 2.2.1. For the position encoding of characters of different granularities, a position-enhanced parallel attention mechanism is used to strengthen the position information of characters. Specifically, a feature sequence is first constructed, in which each feature is assigned a fixed constant 1 / L in its corresponding position index dimension and is set to zero in other dimensions. Here, L represents the total length of the input text. Then, sine and cosine functions are used to encode the position. The specific formula is as follows:

[0029] ;

[0030] ;

[0031] in, Indicates location information. is the position of the character, The dimension of the representation vector is introduced, and a position reinforcement strategy based on self-attention is introduced. Through the back propagation of the model, more targeted position embedding is achieved when decoding different granularities, and the position information is dynamically integrated with the visual information to achieve better recognition accuracy. At the same time, the upper triangular mask Mask is applied to the query vector. Finally, the position clues are injected into the query vector q through two MLP layers. The correlation information is calculated as follows:

[0032] ;

[0033] ;

[0034] in, The function name for multi-head attention mechanism calculation, for The function's parameters, , , denote the weight matrices of Q, K and V respectively, The function represents combining the outputs of different attention heads. is the trainable weight matrix, stands for self-attention network, FFN stands for feed-forward network; finally, the features with enhanced position information are obtained through multi-layer perceptron, residual connection and layer normalization. ; Represents the nth attention head in the multi-head self-attention mechanism;

[0035] Step 2.2.2. Features enhanced using location information As query and initial visual features Perform cross attention; aiming to use the character positions previously decoded at different granularities to search for the character area to be recognized in the text image; specifically, the query Q is the position feature enhanced by the position enhancement module , K and V use the initial visual features extracted by Transformer using a sliding window , initial visual features Visual features of different granularities are obtained through cross-attention modules, multi-layer perceptrons, residual connections, and layer normalization. The cross-attention module is:

[0036] F v i1 =Atten [F POS , Z m ] +FFN (i1=char,cluster,syllable) ;

[0037] Among them, FFN stands for feed-forward network, is the function name for the cross attention mechanism calculation, They represent the extracted visual features of characters, character clusters, and syllables respectively. The cross-attention module extracts and integrates multiple granularity visual features for subsequent multi-granularity decoding.

[0038] Furthermore, the Step 2.3 includes:

[0039] Step 2.3.1, the visual features of different granularities at the character level, character cluster level and syllable level obtained after the multi-granularity feature extraction module driven by language characteristics are defined as [ ],in , used for subsequent multi-granularity feature decoding;

[0040] Step 2.3.2, through the linear layer network, the visual features of different granularities [ ] is transcribed into characters of respective granularity and their confidence. The confidence is calculated as shown in the following formula:

[0041] ( )

[0042] in, Indicates the confidence of decoding at different granularities, is a linear layer network function, and the final result is a collection of confidences of different granularities = { ... };

[0043] Step 2.3.3, the classification results of multi-granularity prediction are generated by different classification heads, and a confidence fusion-based strategy is used to merge and predict the final text results; specifically, the recognition confidence of each character, character cluster and syllable is used to generate the final recognition score through the fusion function. The specific formula of the recognition score is as follows:

[0044] ( );

[0045] in, It represents the confidence of decoding at different granularities. The three recognition scores of the three classification heads of the text image are obtained by accumulating the confidence of each character, and the text result with the highest recognition score is selected as the final prediction result.

[0046] Furthermore, in the Step 2.3, the prediction results are obtained through classification heads of different granularities. The cross entropy loss function is used as the target optimization function of the Burmese text recognition model during model training. The calculation method is shown in the following formula:

[0047] ;

[0048] ;

[0049] in, is the input Burmese text image, is the model parameter of the current recognition network, For Burmese text image The labels corresponding to the feature sequences are is a hyperparameter used to balance the loss, , and represents the hyperparameter used to balance the loss.

[0050] Furthermore, the Step 3 includes:

[0051] The input parameters of the Burmese text image recognition model based on multi-granularity decoding constraints were optimized to achieve batch processing of image data, and the trained converged model was encapsulated for deployment on the server. On this basis, a set of API interfaces were built to achieve efficient recognition of Burmese text images.

[0052] The present invention also provides a Burmese image text recognition device based on multi-granularity decoding constraints, comprising a module for executing the Burmese image text recognition method based on multi-granularity decoding constraints.

[0053] The beneficial effects of the present invention are:

[0054] 1. The present invention first uses a sliding window-based Transformer image feature extraction network to obtain Burmese text image features, and then extracts visual features of characters, character clusters and syllable granularity through a multi-granularity feature extraction module driven by language characteristics. Features of different granularities are then input into a multi-granularity feature fusion module for decoding. Finally, the decoding results are constrained through a confidence fusion strategy, thereby implicitly modeling the language information and improving the recognition accuracy.

[0055] 2. The present invention provides a multi-granularity feature extraction method driven by language characteristics. On the basis of the original character features, visual features of character cluster sequences and syllable sequence features with language knowledge are introduced. On the basis of ensuring the basic recognition accuracy at the character level, the prediction of character cluster level or syllable level granularity is used as a supplement to the noise image prediction results, thereby enhancing the ability to distinguish similar combined characters.

[0056] 3. The present invention uses a visual tranformer structure to replace the traditional CNN structure. This structure can model the global information of the entire image by performing global attention calculation on visual features. At the same time, the recognition results are decoded and constrained based on the confidence fusion strategy, which greatly improves the recognition efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 This is a diagram of the image feature encoding network structure in the present invention;

[0058] Figure 2 This is a structural diagram of the Burmese image text recognition model based on multi-granularity decoding constraints in the present invention;

[0059] Figure 3 This is a structural diagram of the multi-granularity feature extraction module driven by language characteristics in the present invention. DETAILED DESCRIPTION

[0060] Example 1: Figure 1-Figure 3 As shown, the Burmese image text recognition method based on multi-granularity decoding constraints, the specific steps of the method are as follows:

[0061] Step 1, Burmese text image dataset construction and preprocessing: In view of the lack of public Burmese text image datasets in the current field, the present invention constructs a Burmese text image dataset through manual annotation, algorithm generation, etc.; specifically includes:

[0062] Step 1.1, collect Burmese text data from the Internet, and then perform deduplication, segmentation, and special character filtering preprocessing steps to build a Burmese text label dataset for subsequent Burmese text image generation; in addition, in order to enrich the diversity and authenticity of the dataset, obtain Burmese text images of real scenes through the Internet;

[0063] Step 1.2, using the text editing network, replace or modify the text in the natural image without destroying the realism of the original image, so as to generate a text image; specifically, the text image is generated through the text conversion module, the background repair module and the fusion module;

[0064] The text conversion module is used to combine the semantics of the target text with the style of the text in the source image, including font, color, position and scale. The module introduces a skeleton-guided learning mechanism to guide the transfer of text style by predicting the text skeleton graph.

[0065] The background restoration module is responsible for erasing the original text and filling in appropriate textures. It adopts the U-Net structure and realizes background reconstruction through the encoder-decoder architecture.

[0066] The fusion module is used to effectively fuse the output information of the text conversion module and the background restoration module to generate the final edited text image, and divide the training set, test set and verification set according to the final image.

[0067] Step 2: Build a Burmese text image recognition model based on multi-granularity decoding constraints, including:

[0068] Step 2.1. Design a sliding window-based Transformer to extract Burmese text image features and obtain original visual features.

[0069] Furthermore, the Step 2.1 includes:

[0070] Construct a sliding window-based Transformer as the backbone network, such as Figure 1 As shown in the figure, it is used to extract features of the input Burmese text image and convert the image into a token sequence; specifically, the original image is firstly converted into a token sequence through the patch segmentation module. The image is divided into blocks of size P×P, and then a linear embedding layer is used to project the original value feature into the dimension , and flattened into a two-dimensional vector ,in is the resolution of each tile, yes The number of channels, Represents the length of the image block sequence, H, W, and C are the height, width, and number of channels of the image respectively; the token sequence is then sent to the intra-window self-attention W-MSA and inter-window self-attention SW-MSA for modeling. W-MSA and SW-MSA can perform self-attention modeling on the image while maintaining the ability of local perception and global perception. The specific calculation process is as follows:

[0071] ;

[0072] ;

[0073] In the formula, and They represent the outputs of the W-MSA module and the SW-MSA module respectively; Represents the normalization layer, the self-attention-based representation learning calculation formula in the sliding window Transformer is:

[0074] ;

[0075] in, represents the query matrix; represents the bond matrix; represents the value matrix; represents the bias matrix, d1 represents the dimension of the image feature matrix, S represents the number of layers of the current network module, and the encoder feeds the Burmese image into the stacked sliding window Transformer. After multiple stages of network learning, the original visual features are finally obtained. Used for subsequent text recognition.

[0076] Step 2.2, in order to make the model fully utilize visual information of different granularities during the recognition process, a multi-granularity feature extraction module driven by language characteristics is designed: it is used to adaptively extract visual features of different granularities from the original visual features, and align these features with the character positions of the corresponding granularity to support the subsequent multi-granularity decoding process; wherein, the multi-granularity feature extraction module driven by language characteristics is used to capture visual features of different granularities at the character level, character cluster level, and syllable level respectively;

[0077] Furthermore, in the Step 2.2, the multi-granularity feature extraction module driven by the design language characteristics includes two parts: position information enhancement and multi-granularity visual information extraction, such as Figure 3 As shown, it is used to extract the original visual features from Features of different granularities are adaptively extracted and these features are aligned with the character positions of the corresponding granularity, thereby improving the representation ability of text areas in Burmese text images. At the same time, in view of the different position information paid attention to at the same time during decoding at different granularities, a multi-branch visual information extraction module is designed to focus on the visual features of characters, character clusters and syllable granularities respectively for subsequent multi-granularity decoding.

[0078] Furthermore, the Step 2.2 includes:

[0079] Step 2.2.1. For the position encoding of characters of different granularities, a position-enhanced parallel attention mechanism is used to strengthen the position information of characters. Specifically, a feature sequence is first constructed, in which each feature is assigned a fixed constant 1 / L in its corresponding position index dimension and is set to zero in other dimensions. Here, L represents the total length of the input text. Then, sine and cosine functions are used to encode the position. The specific formula is as follows:

[0080] ;

[0081] ;

[0082] in, Indicates location information. is the position of the character, Represents the dimension of the vector. In order to more accurately align the 2D visual features of different granularities and the 1D position embeddings of the corresponding character granularity, a position reinforcement strategy based on self-attention is introduced. Through the back propagation of the model, more targeted position embedding is achieved when decoding different granularities. The position information is dynamically fused with the visual information to achieve better recognition accuracy. At the same time, the upper triangular mask Mask is applied to the query vector to prevent information leakage across time steps. Finally, the position clues are injected into the query vector q through two MLP layers. The correlation information is calculated as follows:

[0083] ;

[0084] ;

[0085] in, The function name for multi-head attention mechanism calculation, for The function's parameters, , , denote the weight matrices of Q, K and V respectively, The function represents combining the outputs of different attention heads. is the trainable weight matrix, stands for self-attention network, FFN stands for feed-forward network; finally, the features with enhanced position information are obtained through multi-layer perceptron, residual connection and layer normalization. ; Represents the nth attention head in the multi-head self-attention mechanism;

[0086] Step 2.2.2: In order to extract visual features of different granularities, use features enhanced by location information As query and initial visual features Perform cross attention; aiming to use the character positions previously decoded at different granularities to search for the character area to be recognized in the text image; specifically, the query Q is the position feature enhanced by the position enhancement module , K and V use the initial visual features extracted by Transformer using a sliding window , initial visual features Visual features of different granularities are obtained through cross-attention modules, multi-layer perceptrons, residual connections, and layer normalization. The cross-attention module is:

[0087] F v i1 =Atten [F POS , Z m ] +FFN (i1=char,cluster,syllable) ;

[0088] Among them, FFN stands for feed-forward network, is the function name for the cross attention mechanism calculation, They represent the extracted visual features of characters, character clusters, and syllables respectively. The cross-attention module extracts and integrates multiple granularity visual features for subsequent multi-granularity decoding.

[0089] Step 2.3, design a multi-granularity feature fusion module to capture visual features of different granularities, use a confidence fusion strategy to perform decoding constraints, and obtain the final predicted text result;

[0090] The Step 2.3 includes:

[0091] Step 2.3.1, the visual features of different granularities at the character level, character cluster level and syllable level obtained after the multi-granularity feature extraction module driven by language characteristics are defined as [ ],in , used for subsequent multi-granularity feature decoding;

[0092] Step 2.3.2, through the linear layer network, the visual features of different granularities [ ] is transcribed into characters of respective granularity and their confidence. The confidence is calculated as shown in the following formula:

[0093] ( )

[0094] in, Indicates the confidence of decoding at different granularities, is a linear layer network function, and the final result is a collection of confidences of different granularities = { ... };

[0095] Step 2.3.3, the classification results of multi-granularity prediction are generated by different classification heads, and a confidence fusion-based strategy is used to merge and predict the final text results; specifically, the recognition confidence of each character, character cluster and syllable is used to generate the final recognition score through the fusion function. The specific formula of the recognition score is as follows:

[0096] ( );

[0097] in, It represents the confidence of decoding at different granularities. The three recognition scores of the three classification heads of the text image are obtained by accumulating the confidence of each character, and the text result with the highest recognition score is selected as the final prediction result.

[0098] In the Step 2.3, the prediction results are obtained through classification heads of different granularities. The cross entropy loss function is used as the target optimization function of the Burmese text recognition model during model training. The calculation method is shown in the following formula:

[0099] ;

[0100] ;

[0101] in, is the input Burmese text image, is the model parameter of the current recognition network, For Burmese text image The labels corresponding to the feature sequences are is a hyperparameter used to balance the loss, , and represents the hyperparameter used to balance the loss.

[0102] Step 3: Use the built Burmese text image recognition model based on multi-granularity decoding constraints to perform Burmese text image recognition. In order to improve the efficiency of Burmese text image recognition, the input parameters of the built Burmese text image recognition model based on multi-granularity decoding constraints were optimized to achieve batch processing of image data, and the trained converged model was packaged for deployment on the server. On this basis, a set of API interfaces was built to achieve efficient recognition of Burmese text images.

[0103] The present invention also provides a Burmese image text recognition device based on multi-granularity decoding constraints, comprising:

[0104] The first building module is used to build the Burmese text image dataset and preprocessing;

[0105] The second building block is used to build a Burmese text image recognition model based on multi-granularity decoding constraints, including:

[0106] Design a sliding window-based Transformer to extract Burmese text image features and obtain original visual features;

[0107] Design a multi-granularity feature extraction module driven by language characteristics: It is used to adaptively extract visual features of different granularities from the original visual features and align these features with the character positions of the corresponding granularity to support the subsequent multi-granularity decoding process; the multi-granularity feature extraction module driven by language characteristics is used to capture visual features of different granularities at the character level, character cluster level, and syllable level respectively;

[0108] A multi-granularity feature fusion module is designed to capture visual features of different granularities, and a confidence fusion strategy is used to perform decoding constraints to obtain the final predicted text result.

[0109] The recognition module is used to perform Burmese text image recognition using a Burmese text image recognition model based on multi-granularity decoding constraints.

[0110] In order to verify the effect of the Burmese image text recognition model based on multi-granularity decoding constraints proposed in this paper, experiments were conducted on 8.4 million self-constructed Burmese text images. About 30,000 of them were manually annotated, and the rest of the data was synthesized by the algorithm. To ensure fairness, we set unified experimental conditions for all comparative experiments. The input image size was 32×128, the optimizer was Adam, the initial learning rate was set to 1, and the CosineAnnealing strategy was used to gradually reduce the learning rate to ensure that the model has a smaller learning rate when it is most solvable. The batch size was set to 128 and trained for 10 epochs.

[0111] The following comparative experiment was designed for analysis.

[0112] The experiment uses Burmese Sequence Accuracy (SA) as the evaluation indicator, as shown below:

[0113]

[0114] Among them, SA, SL, and LN represent the sequence accuracy, the total number of correct sequences, and the total number of sequences of Burmese text image recognition, respectively.

[0115] Experiment 1: Mainly compare the experimental results;

[0116] Table 1 Main experimental results

[0117] As shown in Table 1, the recognition model proposed in the present invention has an accuracy of 97.9% and 97.5% in Burmese and Vietnamese, respectively, as shown in Table 3, achieving the best results. Compared with the connectionist temporal classification method, the accuracy is improved by 7.5% and 6.4% respectively; compared with the sequence-to-sequence method, the accuracy is improved by 7.3% and 5.8% respectively. This proves that the method of the present invention significantly improves the ability to extract Burmese image features by using the sliding window Transformer encoding module for global context modeling. Compared with the recognition method of He et al., the accuracy is improved by 0.7%, which shows that the method of the present invention is based on multi-granularity decoding constraints, which not only guarantees the basic recognition accuracy at the character level, but also uses language information as a supplementary result to the noise image in the prediction of character clusters and syllable granularity, thereby improving the recognition accuracy.

[0118] Experiment 2: Multi-granularity feature extraction and cancellation experiment;

[0119] To verify the effectiveness of the multi-granularity feature extraction module, the present invention conducted an ablation experiment on it on the baseline model. The experimental results are shown in Table 2 ("×" represents no fusion, "√" represents fusion):

[0120] Table 2 Experimental results of multi-granularity feature extraction and cancellation

[0121] As can be seen from Table 2, character granularity, character cluster granularity, and syllable granularity respectively represent the model extracting visual features of character granularity, character cluster granularity, and syllable granularity for decoding. From the experimental results, it can be seen that the classification performance of character cluster granularity or syllable granularity is better than the recognition performance of character granularity. These results show that the prediction of character clusters and syllables can improve text recognition performance by implicitly introducing language information. The effectiveness of the proposed method is proved.

[0122] Experiment 3: Comparison of training speed and inference speed of various models;

[0123] In order to verify the performance of the invented model in terms of training and inference speed, performance tests of multiple mainstream models were performed on the same dataset. During the test, the duration of one round of model training of 2000 steps was taken as the training speed, and the trained model was used to perform inference and prediction on the same Burmese text images to obtain the inference speed of the model.

[0124] Table 3 Comparison of model training speed and inference speed

[0125] As shown in Table 3, the training speed and reasoning speed of the recognition model corresponding to the method of the present invention are not much different from those of the "ResNet50+Transformer" model with better performance at present, which shows that the method proposed in the present invention improves the accuracy of model recognition while adding almost no extra time overhead. In addition, although the model of the present invention is not as fast as the "VGG16+BiLSTM+CTC" model in reasoning speed, the model of the present invention has improved the recognition accuracy compared with these models, which proves the effectiveness of the method proposed in the present invention.

[0126] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A Burmese image text recognition method based on multi-granularity decoding constraints, characterized by: The specific steps of the method are as follows: Step 1: Construction and preprocessing of Burmese text image dataset; Step 2: Build a Burmese text image recognition model based on multi-granularity decoding constraints, including: Step 2.

1. Design a sliding window-based Transformer to extract Burmese text image features and obtain original visual features. Step 2.2, design a multi-granularity feature extraction module driven by language characteristics: used to adaptively extract visual features of different granularities from the original visual features, and align these features with the character positions of the corresponding granularity to support the subsequent multi-granularity decoding process; wherein, the multi-granularity feature extraction module driven by language characteristics is used to capture visual features of different granularities at the character level, character cluster level, and syllable level respectively; Step 2.3, design a multi-granularity feature fusion module to capture visual features of different granularities, use a confidence fusion strategy to perform decoding constraints, and obtain the final predicted text result; Step 3: Use the constructed Burmese text image recognition model based on multi-granularity decoding constraints to perform Burmese text image recognition.

2. The method for Burmese image text recognition based on multi-granularity decoding constraints according to claim 1, characterized in that: The specific steps of Step 1 are: Step 1.1, collect Burmese text data from the Internet, and then perform preprocessing steps such as deduplication, segmentation, and filtering of special characters to build a Burmese text label dataset for subsequent generation of Burmese text images; obtain Burmese text images of real scenes through the Internet; Step 1.2, using the text editing network, replace or modify the text in the natural image without destroying the realism of the original image, so as to generate a text image; specifically, the text image is generated through the text conversion module, the background repair module and the fusion module; The text conversion module is used to combine the semantics of the target text with the style of the text in the source image, including font, color, position and scale. The module introduces a skeleton-guided learning mechanism to guide the transfer of text style by predicting the text skeleton graph. The background restoration module is responsible for erasing the original text and filling in appropriate textures. It adopts the U-Net structure and realizes background reconstruction through the encoder-decoder architecture. The fusion module is used to effectively fuse the output information of the text conversion module and the background restoration module to generate the final edited text image, and divide the training set, test set and verification set according to the final image.

3. The method for Burmese image text recognition based on multi-granularity decoding constraints according to claim 1, characterized in that: The Step 2.1 includes: A sliding window-based Transformer is constructed as the backbone network, which is used to extract features from the input Burmese text image and convert the image into a token sequence. Specifically, the original image is firstly transformed into a token sequence through the patch segmentation module. The image is divided into blocks of size P×P, and then a linear embedding layer is used to project the original value feature into the dimension , and flattened into a two-dimensional vector ,in is the resolution of each tile, yes The number of channels, Represents the length of the image block sequence, H, W, and C are the height, width, and number of channels of the image respectively; the token sequence is then sent to the intra-window self-attention W-MSA and inter-window self-attention SW-MSA for modeling. W-MSA and SW-MSA can perform self-attention modeling on the image while maintaining the ability of local perception and global perception. The specific calculation process is as follows: ； ； In the formula, and They represent the outputs of the W-MSA module and the SW-MSA module respectively; Represents the normalization layer, the self-attention-based representation learning calculation formula in the sliding window Transformer is: ； in, represents the query matrix; represents the bond matrix; represents the value matrix; represents the bias matrix, d1 represents the dimension of the image feature matrix, S represents the number of layers of the current network module, and the encoder feeds the Burmese image into the stacked sliding window Transformer. After multiple stages of network learning, the original visual features are finally obtained. Used for subsequent text recognition.

4. The method for Burmese image text recognition based on multi-granularity decoding constraints according to claim 1, characterized in that: In the Step 2.2, the multi-granularity feature extraction module driven by the design language characteristics includes two parts: position information enhancement and multi-granularity visual information extraction, which are used to extract the original visual features. Features of different granularities are adaptively extracted and these features are aligned with the character positions of the corresponding granularity. At the same time, considering that the position information paid attention to at the same time is different when decoding at different granularities, a multi-branch visual information extraction module is designed to focus on the visual features of characters, character clusters and syllable granularities respectively for subsequent multi-granularity decoding.

5. The method for Burmese image text recognition based on multi-granularity decoding constraints according to claim 1 or 4, characterized in that: The Step 2.2 includes: Step 2.2.

1. For the position encoding of characters of different granularities, a position-enhanced parallel attention mechanism is used to strengthen the position information of characters. Specifically, a feature sequence is first constructed, in which each feature is assigned a fixed constant 1 / L in its corresponding position index dimension and is set to zero in other dimensions. Here, L represents the total length of the input text. Then, sine and cosine functions are used to encode the position. The specific formula is as follows: ； ； in, Indicates location information. is the position of the character, The dimension of the representation vector is introduced, and a position reinforcement strategy based on self-attention is introduced. Through the back propagation of the model, more targeted position embedding is achieved when decoding different granularities, and the position information is dynamically integrated with the visual information to achieve better recognition accuracy. At the same time, the upper triangular mask Mask is applied to the query vector. Finally, the position clues are injected into the query vector q through two MLP layers. The correlation information is calculated as follows: ； ； in, The function name for multi-head attention mechanism calculation, for The function's parameters, , , denote the weight matrices of Q, K and V respectively, The function represents combining the outputs of different attention heads. is the trainable weight matrix, stands for self-attention network, FFN stands for feed-forward network; finally, the features with enhanced position information are obtained through multi-layer perceptron, residual connection and layer normalization. ; Represents the nth attention head in the multi-head self-attention mechanism; Step 2.2.

2. Features enhanced using location information As query and initial visual features Perform cross attention; aiming to use the character positions previously decoded at different granularities to search for the character area to be recognized in the text image; specifically, the query Q is the position feature enhanced by the position enhancement module , K and V use the initial visual features extracted by Transformer using a sliding window , initial visual features Visual features of different granularities are obtained through cross-attention modules, multi-layer perceptrons, residual connections, and layer normalization. The cross-attention module is: ； Among them, FFN stands for feed-forward network, is the function name for the cross attention mechanism calculation, They represent the extracted visual features of characters, character clusters, and syllables respectively. The cross-attention module extracts and integrates multiple granularity visual features for subsequent multi-granularity decoding.

6. The method for Burmese image text recognition based on multi-granularity decoding constraints according to claim 1, characterized in that: The Step 2.3 includes: Step 2.3.1, the visual features of different granularities at the character level, character cluster level and syllable level obtained after the multi-granularity feature extraction module driven by language characteristics are defined as [ ],in , used for subsequent multi-granularity feature decoding; Step 2.3.2, through the linear layer network, the visual features of different granularities [ ] is transcribed into characters of respective granularity and their confidence. The confidence is calculated as shown in the following formula: ( ) in, Indicates the confidence of decoding at different granularities, is a linear layer network function, and the final result is a collection of confidences of different granularities = { ... }; Step 2.3.3, the classification results of multi-granularity prediction are generated by different classification heads, and a confidence fusion-based strategy is used to merge and predict the final text results; specifically, the recognition confidence of each character, character cluster and syllable is used to generate the final recognition score through the fusion function. The specific formula of the recognition score is as follows: ( )； in, It represents the confidence of decoding at different granularities. The three recognition scores of the three classification heads of the text image are obtained by accumulating the confidence of each character, and the text result with the highest recognition score is selected as the final prediction result.

7. The method for Burmese image text recognition based on multi-granularity decoding constraints according to claim 1, characterized in that: In the Step 2.3, the prediction results are obtained through classification heads of different granularities. The cross entropy loss function is used as the target optimization function of the Burmese text recognition model during model training. The calculation method is shown in the following formula: ； ； in, is the input Burmese text image, is the model parameter of the current recognition network, For Burmese text image The labels corresponding to the feature sequences are is a hyperparameter used to balance the loss, , and represents the hyperparameter used to balance the loss.

8. The method for Burmese image text recognition based on multi-granularity decoding constraints according to claim 1, characterized in that: The Step 3 includes: The input parameters of the Burmese text image recognition model based on multi-granularity decoding constraints were optimized to achieve batch processing of image data, and the trained converged model was encapsulated for deployment on the server. On this basis, a set of API interfaces were built to achieve efficient recognition of Burmese text images.

9. A Burmese image text recognition device based on multi-granularity decoding constraints, characterized in that: The method comprises a module for executing the Burmese image text recognition method based on multi-granularity decoding constraints as described in any one of claims 1 to 8.

Images