Burmese image text recognition method and device based on multi-granularity decoding constraints

Abstract

The present invention relates to a method and device for Burmese text recognition based on multi-granularity decoding constraints, and belongs to the field of natural language processing. Burmese text has a unique encoding sequence and character combination rules. In order to solve the problem of poor recognition of similar characters in Burmese images, a method for Burmese text image recognition based on multi-granularity decoding constraints is proposed. The method mainly consists of four parts: construction and preprocessing of a Burmese text image dataset, design of a Burmese text image recognition model based on multi-granularity decoding constraints including a Transformer based on a sliding window, a multi-granularity feature extraction module driven by language characteristics, and a multi-granularity feature fusion module; and Burmese text image recognition. According to the functional modularization of these four parts, a Burmese text image recognition device based on multi-granularity decoding constraints is manufactured, which effectively solves the problem of difficulty in recognizing similar Burmese characters due to poor image quality.

Description

Technical Field

[0001] The present invention discloses a Burmese image text recognition method and device based on multi-granularity decoding constraints, belonging to the technical field of natural language processing. Background Art

[0002] Burmese has a unique encoding order and character combination rules. In addition to basic consonant characters, it also includes modifiers such as left-hand vowel characters, upper-hand vowel characters, right-hand vowel characters, and lower-hand tone characters. These modifiers can be nested in different positions of basic characters to form complex character combinations.

[0003] A Burmese syllable may be composed of multiple independently encoded Unicode characters. This nested nature makes Burmese character recognition more complex than linear character recognition methods for languages ​​like Chinese and English. During the recognition process, edge features of characters are often overlooked, leading to misrecognition or loss of some characters during decoding and garbled results. Furthermore, the high similarity between some Burmese characters significantly interferes with Burmese recognition. Similar characters often lead to confusion in recognition results. Therefore, recognition models that capture more fine-grained features can mitigate this problem.

[0004] To address the above issues, the present invention proposes a Burmese image text recognition method and device based on multi-granularity decoding constraints. Based on the original character sequence decoding, character cluster sequences and syllable sequences with linguistic knowledge are introduced to establish decoding constraints, implicitly integrating linguistic knowledge into the decoding process and enhancing the ability to distinguish similar composite characters. At the same time, through a multi-granularity feature extraction module driven by language characteristics, the visual features of characters at different granularities are adaptively captured and better aligned with the position information. Finally, in the inference stage, the predictions of the three granularities are fused to obtain the final recognition result. Summary of the Invention

[0005] The present invention proposes a Burmese image text recognition method and device based on multi-granularity decoding constraints, which is used to solve the problem of poor recognition of similar characters in Burmese images and the difficulty in recognizing similar Burmese characters due to poor image quality. The present invention has a high recognition accuracy.

[0006] The technical solution of the present invention is: a method for Burmese image text recognition based on multi-granularity decoding constraints, the specific steps of the method are as follows:

[0007] Step 1: Construction and preprocessing of Burmese text image dataset;

[0008] Step 2: Build a Burmese text image recognition model based on multi-granularity decoding constraints, including:

[0009] Step 2.1. Design a sliding window-based Transformer to extract Burmese text image features and obtain original visual features.

[0010] Step 2.2: Design a language-feature driven multi-granularity feature extraction module. This module is used to adaptively extract visual features of different granularities from the original visual features and align these features with the character positions of the corresponding granularity to support the subsequent multi-granularity decoding process. The language-feature driven multi-granularity feature extraction module captures visual features of different granularities at the character level, character cluster level, and syllable level.

[0011] Step 2.3: Design a multi-granularity feature fusion module to decode the captured visual features at different granularities using a confidence fusion strategy to obtain the final predicted text result. This process significantly improves the model's recognition accuracy for easily confused and similar characters in Burmese.

[0012] Step 3: Use the constructed Burmese text image recognition model based on multi-granularity decoding constraints to perform Burmese text image recognition.

[0013] Furthermore, the specific steps of Step 1 are:

[0014] Step 1.1: Collect Burmese text data from the internet, perform preprocessing steps such as deduplication, segmentation, and filtering of special characters to construct a Burmese text label dataset for subsequent Burmese text image generation. Real-world Burmese text images are obtained from the internet.

[0015] Step 1.2: Using the text editing network, replace or modify the text in the natural image without destroying the realism of the original image, thereby generating a text image. Specifically, the text image is generated through the text conversion module, background restoration module, and fusion module.

[0016] The text conversion module is used to combine the semantics of the target text with the style of the text in the source image, including font, color, position, and scale. This module introduces a skeleton-guided learning mechanism to guide the transfer of text style by predicting text skeleton graphs.

[0017] The background restoration module is responsible for erasing the original text and filling it with appropriate textures. It adopts the U-Net structure and uses the encoder-decoder architecture to reconstruct the background.

[0018] The fusion module is used to effectively fuse the output information of the text conversion module and the background restoration module to generate the final edited text image, and then divide the training set, test set and validation set according to the final image.

[0019] Furthermore, the Step 2.1 includes:

[0020] Construct a Transformer based on sliding window as the backbone network, use it to extract features of the input Burmese text image, and convert the image into a token sequence; specifically, the original image is first transformed into a token sequence through the patch segmentation module. The image is divided into blocks of size P×P, and then a linear embedding layer is used to project the original value feature into the dimension , and flattened into a two-dimensional vector ,in is the resolution of each tile, yes The number of channels, Represents the length of the image block sequence, H, W, and C are the height, width, and number of channels of the image respectively; the token sequence is then fed into the intra-window self-attention W-MSA and inter-window self-attention SW-MSA for modeling. W-MSA and SW-MSA can perform self-attention modeling on the image while maintaining the ability of local perception and global perception. The specific calculation process is as follows:

[0021] ;

[0022] ;

[0023] Where, and Represent the outputs of the W-MSA module and the SW-MSA module respectively; Represents the normalization layer. The calculation formula for self-attention-based representation learning in the sliding window Transformer is:

[0024] ;

[0025] in, represents the query matrix; represents the bond matrix; represents the value matrix; Represents the bias matrix, d1 represents the dimension of the image feature matrix, and S represents the number of layers in the current network module. The encoder feeds the Burmese image into the stacked sliding window Transformer, and after multiple stages of network learning, the original visual features are finally obtained. Used for subsequent text recognition.

[0026] Furthermore, in Step 2.2, the multi-granularity feature extraction module driven by the design language characteristics includes two parts: position information enhancement and multi-granularity visual information extraction, which is used to extract the original visual features. Features of different granularities are adaptively extracted and these features are aligned with the character positions of the corresponding granularity. At the same time, given that the position information of interest at the same time is different when decoding at different granularities, a multi-branch visual information extraction module is designed to focus on the visual features of characters, character clusters, and syllable granularity respectively for subsequent multi-granularity decoding.

[0027] Furthermore, the Step 2.2 includes:

[0028] Step 2.2.1: For the position encoding of characters of different granularities, a position-enhanced parallel attention mechanism is used to enhance the position information of characters. Specifically, a feature sequence is first constructed, where each feature is assigned a fixed constant 1 / L in its corresponding position index dimension and is set to zero in other dimensions. Here, L represents the total length of the input text. Then, sine and cosine functions are used to encode the position. The specific formula is as follows:

[0029] ;

[0030] ;

[0031] in, Indicates location information. is the position of the character, The dimension of the representation vector is determined by introducing a position reinforcement strategy based on self-attention. Through model backpropagation, more targeted position embedding is achieved when decoding at different granularities. Position information is dynamically integrated with visual information to achieve better recognition accuracy. At the same time, an upper triangular mask is applied to the query vector. Finally, position clues are injected into the query vector q through two MLP layers. The correlation information is calculated as follows:

[0032] ;

[0033] ;

[0034] in, The function name for multi-head attention mechanism calculation, for The formal parameters of the function, , , Denote the weight matrices of Q, K and V respectively, The function represents combining the outputs of different attention heads. is the trainable weight matrix, stands for self-attention network, FFN stands for feedforward network; finally, the position information enhanced feature is obtained through multi-layer perceptron, residual connection and layer normalization ; Represents the nth attention head in the multi-head self-attention mechanism;

[0035] Step 2.2.2. Features enhanced using location information As query and initial visual features Perform cross attention; aiming to use the character positions previously decoded at different granularities to search for the character area to be recognized in the text image; specifically, the query Q is the position feature enhanced by the position enhancement module , K and V use the initial visual features extracted by Transformer using a sliding window , initial visual features Visual features of different granularities are obtained through the cross-attention module, multi-layer perceptron, residual connection and layer normalization. The cross-attention module is:

[0036] F v i1 =Atten [F POS , Z m ] +FFN (i1=char,cluster,syllable) ;

[0037] Among them, FFN stands for feed-forward network, is the function name for the cross attention mechanism calculation, They represent the extracted visual features of characters, character clusters, and syllables respectively. The cross-attention module extracts and integrates multiple granularity visual features for subsequent multi-granularity decoding.

[0038] Furthermore, Step 2.3 includes:

[0039] Step 2.3.1, the visual features of different granularities at the character level, character cluster level and syllable level obtained after the language feature driven multi-granularity feature extraction module are defined as [ ],in , used for subsequent multi-granularity feature decoding;

[0040] Step 2.3.2, through the linear layer network, the visual features of different granularities [ ] is transcribed into characters of respective granularity and their confidence. The confidence is calculated as shown in the following formula:

[0041] ( )

[0042] in, Indicates the confidence of decoding at different granularities, is a linear layer network function, and the final result is a collection of confidences of different granularities = { ... };

[0043] Step 2.3.3: The classification results of multi-granularity predictions are generated by different classification heads, and a confidence fusion-based strategy is used to merge and predict the final text results. Specifically, the recognition confidence of each character, character cluster, and syllable is fused through a function to generate the final recognition score. The specific formula for the recognition score is as follows:

[0044] ( );

[0045] in, It represents the confidence of decoding at different granularities. The three recognition scores of the three classification heads of the text image are obtained by accumulating the confidence of each character. The text result with the highest recognition score is selected as the final prediction result.

[0046] Furthermore, in Step 2.3, the prediction results are obtained through classification heads of different granularities. During model training, the cross entropy loss function is used as the target optimization function of the Burmese text recognition model. The calculation method is shown in the following formula:

[0047] ;

[0048] ;

[0049] in, is the input Burmese text image, is the model parameter of the current recognition network, For Burmese text image The labels corresponding to the feature sequences, is a hyperparameter used to balance the loss, 、 and represents the hyperparameter used to balance the loss.

[0050] Furthermore, the Step 3 includes:

[0051] The input parameters of the Burmese text image recognition model based on multi-granularity decoding constraints were optimized to achieve batch processing of image data. The trained and converged model was packaged for deployment on the server. On this basis, a set of API interfaces was built to achieve efficient recognition of Burmese text images.

[0052] The present invention also provides a Burmese image text recognition device based on multi-granularity decoding constraints, comprising a module for executing the Burmese image text recognition method based on multi-granularity decoding constraints.

[0053] The beneficial effects of the present invention are:

[0054] 1. This paper first uses a sliding window-based Transformer image feature extraction network to obtain Burmese text image features. Then, a language-specific multi-granularity feature extraction module is used to extract visual features at the character, character cluster, and syllable granularity levels. The features at different granularities are then input into a multi-granularity feature fusion module for decoding. Finally, a confidence fusion strategy is used to constrain the decoding results, thereby implicitly modeling the language information and improving recognition accuracy.

[0055] 2. The present invention provides a multi-granularity feature extraction method driven by language characteristics. On the basis of the original character features, it introduces visual features of character cluster sequence and syllable sequence features with language knowledge. On the basis of ensuring the basic recognition accuracy at the character level, the prediction of character cluster level or syllable level granularity is used as a supplement to the noise image prediction results, thereby enhancing the ability to distinguish similar combination characters.

[0056] 3. This invention replaces the traditional CNN architecture with a visual transformer architecture. This architecture models the global information of the entire image by performing global attention calculations on visual features. Furthermore, a confidence fusion strategy is used to constrain the decoding of recognition results, significantly improving recognition efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 This is a diagram of the image feature encoding network structure in the present invention;

[0058] Figure 2 This is a structural diagram of the Burmese image text recognition model based on multi-granularity decoding constraints in the present invention;

[0059] Figure 3 This is a structural diagram of the multi-granularity feature extraction module driven by language characteristics in the present invention. DETAILED DESCRIPTION

[0060] Example 1: Figure 1-Figure 3 As shown in FIG, a method for Burmese image text recognition based on multi-granularity decoding constraints is described, and the specific steps of the method are as follows:

[0061] Step 1: Construction and preprocessing of Burmese text image dataset. To address the current lack of public Burmese text image datasets, this paper constructs a Burmese text image dataset through manual annotation and algorithm generation. Specifically, it includes:

[0062] Step 1.1: Collect Burmese text data from the internet. Then perform preprocessing steps such as deduplication, segmentation, and filtering of special characters to construct a Burmese text label dataset for subsequent generation of Burmese text images. In addition, to enrich the diversity and authenticity of the dataset, obtain real-world Burmese text images from the internet.

[0063] Step 1.2: Using the text editing network, replace or modify the text in the natural image without destroying the realism of the original image, thereby generating a text image. Specifically, the text image is generated through the text conversion module, background restoration module, and fusion module.

[0064] The text conversion module is used to combine the semantics of the target text with the style of the text in the source image, including font, color, position, and scale. This module introduces a skeleton-guided learning mechanism to guide the transfer of text style by predicting text skeleton graphs.

[0065] The background restoration module is responsible for erasing the original text and filling it with appropriate textures. It adopts the U-Net structure and uses the encoder-decoder architecture to reconstruct the background.

[0066] The fusion module is used to effectively fuse the output information of the text conversion module and the background restoration module to generate the final edited text image, and then divide the training set, test set and validation set according to the final image.

[0067] Step 2: Build a Burmese text image recognition model based on multi-granularity decoding constraints, including:

[0068] Step 2.1. Design a sliding window-based Transformer to extract Burmese text image features and obtain original visual features.

[0069] Furthermore, the Step 2.1 includes:

[0070] Construct a sliding window-based Transformer as the backbone network, such as Figure 1 As shown in the figure, it is used to extract features of the input Burmese text image and convert the image into a token sequence; specifically, the original image is firstly transformed into a token sequence through the patch segmentation module. The image is divided into blocks of size P×P, and then a linear embedding layer is used to project the original value feature into the dimension , and flattened into a two-dimensional vector ,in is the resolution of each tile, yes The number of channels, Represents the length of the image block sequence, H, W, and C are the height, width, and number of channels of the image respectively; the token sequence is then fed into the intra-window self-attention W-MSA and inter-window self-attention SW-MSA for modeling. W-MSA and SW-MSA can perform self-attention modeling on the image while maintaining the ability of local perception and global perception. The specific calculation process is as follows:

[0071] ;

[0072] ;

[0073] Where, and Represent the outputs of the W-MSA module and the SW-MSA module respectively; Represents the normalization layer. The calculation formula for self-attention-based representation learning in the sliding window Transformer is:

[0074] ;

[0075] in, represents the query matrix; represents the bond matrix; represents the value matrix; Represents the bias matrix, d1 represents the dimension of the image feature matrix, and S represents the number of layers in the current network module. The encoder feeds the Burmese image into the stacked sliding window Transformer, and after multiple stages of network learning, the original visual features are finally obtained. Used for subsequent text recognition.

[0076] Step 2.2: To enable the model to fully utilize visual information at different granularities during the recognition process, a language-feature-driven multi-granularity feature extraction module is designed. This module is used to adaptively extract visual features at different granularities from the original visual features and align these features with the character positions at the corresponding granularity to support the subsequent multi-granularity decoding process. The language-feature-driven multi-granularity feature extraction module captures visual features at different granularities, including character level, character cluster level, and syllable level.

[0077] Furthermore, in Step 2.2, the multi-granularity feature extraction module driven by the design language characteristics includes two parts: position information enhancement and multi-granularity visual information extraction, such as Figure 3 As shown, it is used to extract the original visual features from Features of different granularities are adaptively extracted and these features are aligned with the character positions of the corresponding granularity, thereby improving the representation ability of text areas in Burmese text images. At the same time, given that the position information paid attention to at the same time is different when decoding at different granularities, a multi-branch visual information extraction module is designed to focus on the visual features of characters, character clusters and syllable granularity respectively for subsequent multi-granularity decoding.

[0078] Furthermore, the Step 2.2 includes:

[0079] Step 2.2.1: For the position encoding of characters of different granularities, a position-enhanced parallel attention mechanism is used to enhance the position information of characters. Specifically, a feature sequence is first constructed, where each feature is assigned a fixed constant 1 / L in its corresponding position index dimension and is set to zero in other dimensions. Here, L represents the total length of the input text. Then, sine and cosine functions are used to encode the position. The specific formula is as follows:

[0080] ;

[0081] ;

[0082] in, Indicates location information. is the position of the character, To more accurately align the 2D visual features of different granularities with the 1D position embeddings of the corresponding character granularity, a position reinforcement strategy based on self-attention is introduced. Model backpropagation is used to achieve more targeted position embedding when decoding different granularities, dynamically fusing position information with visual information to achieve better recognition accuracy. At the same time, an upper triangular mask is applied to the query vector to prevent information leakage across time steps. Finally, position cues are injected into the query vector q through two MLP layers. The correlation information is calculated as follows:

[0083] ;

[0084] ;

[0085] in, The function name for multi-head attention mechanism calculation, for The formal parameters of the function, , , Denote the weight matrices of Q, K and V respectively, The function represents combining the outputs of different attention heads. is the trainable weight matrix, stands for self-attention network, FFN stands for feedforward network; finally, the position information enhanced feature is obtained through multi-layer perceptron, residual connection and layer normalization ; Represents the nth attention head in the multi-head self-attention mechanism;

[0086] Step 2.2.2: In order to extract visual features of different granularities, use features enhanced by position information As query and initial visual features Perform cross attention; aiming to use the character positions previously decoded at different granularities to search for the character area to be recognized in the text image; specifically, the query Q is the position feature enhanced by the position enhancement module , K and V use the initial visual features extracted by Transformer using a sliding window , initial visual features Visual features of different granularities are obtained through the cross-attention module, multi-layer perceptron, residual connection and layer normalization. The cross-attention module is:

[0087] F v i1 =Atten [F POS , Z m ] +FFN (i1=char,cluster,syllable) ;

[0088] Among them, FFN stands for feed-forward network, is the function name for the cross attention mechanism calculation, They represent the extracted visual features of characters, character clusters, and syllables respectively. The cross-attention module extracts and integrates multiple granularity visual features for subsequent multi-granularity decoding.

[0089] Step 2.3: Design a multi-granularity feature fusion module to decode the captured visual features of different granularities using a confidence fusion strategy to obtain the final predicted text result.

[0090] Step 2.3 includes:

[0091] Step 2.3.1, the visual features of different granularities at the character level, character cluster level and syllable level obtained after the language feature driven multi-granularity feature extraction module are defined as [ ],in , used for subsequent multi-granularity feature decoding;

[0092] Step 2.3.2, through the linear layer network, the visual features of different granularities [ ] is transcribed into characters of respective granularity and their confidence. The confidence is calculated as shown in the following formula:

[0093] ( )

[0094] in, Indicates the confidence of decoding at different granularities, is a linear layer network function, and the final result is a collection of confidences of different granularities = { ... };

[0095] Step 2.3.3: The classification results of multi-granularity predictions are generated by different classification heads, and a confidence fusion-based strategy is used to merge and predict the final text results. Specifically, the recognition confidence of each character, character cluster, and syllable is fused through a function to generate the final recognition score. The specific formula for the recognition score is as follows:

[0096] ( );

[0097] in, It represents the confidence of decoding at different granularities. The three recognition scores of the three classification heads of the text image are obtained by accumulating the confidence of each character. The text result with the highest recognition score is selected as the final prediction result.

[0098] In Step 2.3, the prediction results are obtained through classification heads of different granularities. The cross entropy loss function is used as the target optimization function of the Burmese text recognition model during model training. The calculation method is shown in the following formula:

[0099] ;

[0100] ;

[0101] in, is the input Burmese text image, is the model parameter of the current recognition network, For Burmese text image The labels corresponding to the feature sequences, is a hyperparameter used to balance the loss, 、 and represents the hyperparameter used to balance the loss.

[0102] Step 3: Use the established Burmese text image recognition model based on multi-granularity decoding constraints to perform Burmese text image recognition. To improve the efficiency of Burmese text image recognition, the input parameters of the established Burmese text image recognition model based on multi-granularity decoding constraints were optimized to enable batch processing of image data. The trained and converged model was packaged for deployment on the server. Based on this, a set of API interfaces was built to achieve efficient recognition of Burmese text images.

[0103] The present invention also provides a Burmese image text recognition device based on multi-granularity decoding constraints, comprising:

[0104] The first building block is used to construct and preprocess the Burmese text image dataset;

[0105] The second building block is used to build a Burmese text image recognition model based on multi-granularity decoding constraints, including:

[0106] Design a sliding window-based Transformer to extract Burmese text image features and obtain original visual features;

[0107] Design a language-feature driven multi-granularity feature extraction module: This module is used to adaptively extract visual features of different granularities from the original visual features and align these features with the character positions of the corresponding granularity to support the subsequent multi-granularity decoding process. Specifically, the language-feature driven multi-granularity feature extraction module is used to capture visual features of different granularities at the character level, character cluster level, and syllable level.

[0108] A multi-granularity feature fusion module is designed to capture visual features of different granularities and use a confidence fusion strategy to perform decoding constraints to obtain the final predicted text result.

[0109] The recognition module is used to perform Burmese text image recognition using the constructed Burmese text image recognition model based on multi-granularity decoding constraints.

[0110] To validate the effectiveness of the Burmese image text recognition model proposed in this paper based on multi-granularity decoding constraints, experiments were conducted on 8.4 million self-constructed Burmese text images. Approximately 30,000 of these images were manually annotated, and the remaining data was synthesized algorithmically. To ensure fairness, we used uniform experimental conditions for all comparative experiments: the input image size was 32×128, the Adam optimizer was used, the initial learning rate was set to 1, and the learning rate was gradually reduced using the CosineAnnealing strategy to ensure a smaller learning rate when the model has the best solution. The batch size was set to 128, and training was performed for 10 epochs.

[0111] The following comparative experiment was designed for analysis.

[0112] The experiment uses Burmese Sequence Accuracy (SA) as the evaluation metric, as shown below:

[0113]

[0114] Among them, SA, SL, and LN represent the sequence accuracy, the total number of correct sequences, and the total number of sequences of Burmese text image recognition, respectively.

[0115] Experiment 1: Main comparative experimental results;

[0116] Table 1 Main experimental results

[0117]

[0118] As shown in Table 1, and Table 3, the recognition model proposed by the present invention achieved an accuracy of 97.9% and 97.5% for Burmese and Vietnamese, respectively, achieving optimal results. Compared with the connectionist temporal classification method, the accuracy was improved by 7.5% and 6.4%, respectively; compared with the sequence-to-sequence method, the accuracy was improved by 7.3% and 5.8%, respectively. This proves that the method of the present invention significantly improves the ability to extract Burmese image features by using a sliding window Transformer encoding module for global context modeling. Compared with the recognition method of He et al., the accuracy was improved by 0.7%, which shows that the method of the present invention, based on multi-granularity decoding constraints, not only guarantees basic recognition accuracy at the character level, but also uses language information as a supplement to the noise image in character cluster and syllable granularity prediction, thereby improving recognition accuracy.

[0119] Experiment 2: Multi-granularity feature extraction and cancellation experiment;

[0120] To verify the effectiveness of the multi-granularity feature extraction module, we conducted an ablation experiment on the baseline model. The experimental results are shown in Table 2 ("×" indicates non-integration, "√" indicates integration):

[0121] Table 2. Experimental results of multi-granularity feature extraction and cancellation

[0122]

[0123] As shown in Table 2, character granularity, character cluster granularity, and syllable granularity represent the model's extraction of visual features at the character, character cluster, and syllable granularity levels, respectively, for decoding. Experimental results show that classification performance at the character cluster or syllable granularity outperforms recognition performance at the character granularity level. These results demonstrate that character cluster and syllable prediction can improve text recognition performance by implicitly incorporating linguistic information, demonstrating the effectiveness of the proposed method.

[0124] Experiment 3: Comparison of training speed and inference speed of various models;

[0125] To verify the performance of the invented model in terms of training and inference speed, performance tests of multiple mainstream models were conducted on the same dataset. During the test, the duration of one round of model training of 2000 steps was taken as the training speed, and the trained model was used to perform inference and prediction on the same Burmese text images to obtain the model's inference speed.

[0126] Table 3 Comparison of model training speed and inference speed

[0127]

[0128] As shown in Table 3, the training and inference speeds of the recognition model corresponding to the present method are similar to those of the currently high-performing "ResNet50+Transformer" model, demonstrating that the proposed method improves model recognition accuracy while adding almost no additional time overhead. Furthermore, while the inference speed of the proposed model is inferior to that of models such as "VGG16+BiLSTM+CTC," the proposed model improves recognition accuracy compared to these models, demonstrating the effectiveness of the proposed method.

[0129] The specific embodiments of the present invention are described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Various changes can be made within the knowledge of ordinary technicians in this field without departing from the scope of the present invention.

Claims

1. A Burmese image text recognition method based on multi-granularity decoding constraints, characterized by: The specific steps of the method are as follows: Step 1: Construction and preprocessing of Burmese text image dataset; Step 2: Build a Burmese text image recognition model based on multi-granularity decoding constraints, including: Step 2.

1. Design a sliding window-based Transformer to extract Burmese text image features and obtain original visual features. Step 2.2: Design a language-feature driven multi-granularity feature extraction module. This module is used to adaptively extract visual features of different granularities from the original visual features and align these features with the character positions of the corresponding granularity to support the subsequent multi-granularity decoding process. The language-feature driven multi-granularity feature extraction module captures visual features of different granularities at the character level, character cluster level, and syllable level. Step 2.3: Design a multi-granularity feature fusion module to decode the captured visual features of different granularities using a confidence fusion strategy to obtain the final predicted text result. Step 3: Use the constructed Burmese text image recognition model based on multi-granularity decoding constraints to perform Burmese text image recognition; In Step 2.2, the multi-granularity feature extraction module driven by the design language characteristics includes two parts: position information enhancement and multi-granularity visual information extraction, which is used to extract the original visual features. In this paper, we adaptively extract features of different granularities and align these features with the character positions of the corresponding granularity. At the same time, considering that different granularity decoding focuses on different position information at the same time, we design a multi-branch visual information extraction module to focus on the visual features of characters, character clusters, and syllable granularity respectively for subsequent multi-granularity decoding. Step 2.3 includes: Step 2.3.1, the visual features of different granularities at the character level, character cluster level and syllable level obtained after the language feature driven multi-granularity feature extraction module are defined as [ ],in , used for subsequent multi-granularity feature decoding, Represent the extracted visual features of characters, character clusters, and syllables respectively; Step 2.3.2, through the linear layer network, the visual features of different granularities [ ] is transcribed into characters of respective granularity and their confidence. The confidence is calculated as shown in the following formula: ( ) in, Indicates the confidence of decoding at different granularities, is a linear layer network function, and the final result is a collection of confidences of different granularities = { ... }; Step 2.3.3: The classification results of multi-granularity predictions are generated by different classification heads, and a confidence fusion-based strategy is used to merge and predict the final text results. Specifically, the recognition confidence of each character, character cluster, and syllable is fused through a function to generate the final recognition score. The specific formula for the recognition score is as follows: ( )； in, It represents the confidence of decoding at different granularities. The three recognition scores of the three classification heads of the text image are obtained by accumulating the confidence of each character. The text result with the highest recognition score is selected as the final prediction result.

2. The Burmese image text recognition method based on multi-granularity decoding constraints according to claim 1, characterized in that: The specific steps of Step 1 are: Step 1.1: Collect Burmese text data from the internet, perform preprocessing steps such as deduplication, segmentation, and filtering of special characters to construct a Burmese text label dataset for subsequent Burmese text image generation. Real-world Burmese text images are obtained from the internet. Step 1.2: Using the text editing network, replace or modify the text in the natural image without destroying the realism of the original image, thereby generating a text image. Specifically, the text image is generated through the text conversion module, background restoration module, and fusion module. The text conversion module is used to combine the semantics of the target text with the style of the text in the source image, including font, color, position, and scale. This module introduces a skeleton-guided learning mechanism to guide the transfer of text style by predicting text skeleton graphs. The background restoration module is responsible for erasing the original text and filling it with appropriate textures. It adopts the U-Net structure and uses the encoder-decoder architecture to reconstruct the background. The fusion module is used to effectively fuse the output information of the text conversion module and the background restoration module to generate the final edited text image, and then divide the training set, test set and validation set according to the final image.

3. The Burmese image text recognition method based on multi-granularity decoding constraints according to claim 1, characterized in that: Step 2.1 includes: Construct a Transformer based on sliding window as the backbone network, use it to extract features of the input Burmese text image, and convert the image into a token sequence; specifically, the original image is first transformed into a token sequence through the patch segmentation module. The image is divided into blocks of size P×P, and then a linear embedding layer is used to project the original value feature into the dimension , and flattened into a two-dimensional vector ,in is the resolution of each tile, yes The number of channels, Represents the length of the image block sequence, H, W, and C are the height, width, and number of channels of the image respectively; the token sequence is then fed into the intra-window self-attention W-MSA and inter-window self-attention SW-MSA for modeling. W-MSA and SW-MSA can perform self-attention modeling on the image while maintaining the ability of local perception and global perception. The specific calculation process is as follows: ； ； Where, and Represent the outputs of the W-MSA module and the SW-MSA module respectively; Represents the normalization layer. The calculation formula for self-attention-based representation learning in the sliding window Transformer is: ； in, represents the query matrix; represents the bond matrix; represents the value matrix; Represents the bias matrix, d1 represents the dimension of the image feature matrix, and S represents the number of layers in the current network module. The encoder feeds the Burmese image into the stacked sliding window Transformer, and after multiple stages of network learning, the original visual features are finally obtained. Used for subsequent text recognition.

4. The method for Burmese image text recognition based on multi-granularity decoding constraints according to claim 1, characterized in that: Step 2.2 includes: Step 2.2.1: For the position encoding of characters of different granularities, a position-enhanced parallel attention mechanism is used to enhance the position information of characters. Specifically, a feature sequence is first constructed, where each feature is assigned a fixed constant 1 / L in its corresponding position index dimension and is set to zero in other dimensions. Here, L represents the total length of the input text. Then, sine and cosine functions are used to encode the position. The specific formula is as follows: ； ； in, Indicates location information. is the position of the character, The dimension of the representation vector is determined by introducing a position reinforcement strategy based on self-attention. Through model backpropagation, more targeted position embedding is achieved when decoding at different granularities. Position information is dynamically integrated with visual information to achieve better recognition accuracy. At the same time, an upper triangular mask is applied to the query vector. Finally, position clues are injected into the query vector q through two MLP layers. The correlation information is calculated as follows: ； ； in, The function name for multi-head attention mechanism calculation, for The formal parameters of the function, , , Denote the weight matrices of Q, K and V respectively, The function represents combining the outputs of different attention heads. is the trainable weight matrix, stands for self-attention network, FFN stands for feedforward network; finally, the position information enhanced feature is obtained through multi-layer perceptron, residual connection and layer normalization ; Represents the nth attention head in the multi-head self-attention mechanism; Step 2.2.

2. Features enhanced using location information As query and initial visual features Perform cross attention; aiming to use the character positions previously decoded at different granularities to search for the character area to be recognized in the text image; specifically, the query Q is the position feature enhanced by the position enhancement module , K and V use the initial visual features extracted by Transformer using a sliding window , initial visual features Visual features of different granularities are obtained through the cross-attention module, multi-layer perceptron, residual connection and layer normalization. The cross-attention module is: ； Among them, FFN stands for feed-forward network, It is the name of the function that calculates the cross-attention mechanism. The cross-attention module extracts and integrates multiple granularity visual features for subsequent multi-granularity decoding.

5. The method for Burmese image text recognition based on multi-granularity decoding constraints according to claim 1, characterized in that: In Step 2.3, the prediction results are obtained through classification heads of different granularities. The cross entropy loss function is used as the target optimization function of the Burmese text recognition model during model training. The calculation method is shown in the following formula: ； ； in, is the input Burmese text image, is the model parameter of the current recognition network, For Burmese text image The labels corresponding to the feature sequences, is a hyperparameter used to balance the loss, 、 and represents the hyperparameter used to balance the loss.

6. The method for Burmese image text recognition based on multi-granularity decoding constraints according to claim 1, characterized in that: Step 3 includes: The input parameters of the Burmese text image recognition model based on multi-granularity decoding constraints were optimized to achieve batch processing of image data. The trained and converged model was packaged for deployment on the server. On this basis, a set of API interfaces was built to achieve efficient recognition of Burmese text images.

7. A Burmese image text recognition device based on multi-granularity decoding constraints, characterized in that: The method comprises a module for executing the Burmese image text recognition method based on multi-granularity decoding constraints as described in any one of claims 1 to 6.

Images