Laotian image text recognition method and device integrating language error correction model

Abstract

The present invention relates to a Laotian image text recognition method and device that incorporates a language error correction model, and belongs to the field of natural language processing. Laotian character structures are more complex and contain numerous homographs. To address the problem of poor recognition of similar characters in Laotian images, the present invention is proposed. The present invention comprises four parts: constructing a Laotian text image dataset, constructing a Laotian image text recognition model that incorporates a language error correction model, training a Laotian image text recognition model that incorporates a language error correction model, and recognizing Laotian text images. A Laotian image text recognition device that incorporates a language error correction model is modularized based on the functions of these four parts, effectively addressing the difficulty in Laotian character recognition due to similar glyph structures.

Description

Technical Field

[0001] The present invention provides a Laotian image text recognition method and device integrating a language error correction model, belonging to the technical field of natural language processing. Background Art

[0002] Lao has a unique encoding sequence and character combination rules. The Lao character structure is more complex, using a top-down, left-right arrangement of vowels, consonants, and tone symbols. The relative positioning of characters is complex, far exceeding the linear arrangement of English. Furthermore, Lao characters often lack clear spacing, and connected or overlapping characters are common, leading to difficulties in character segmentation and recognition. Furthermore, Lao has many homographs, making individual characters difficult to accurately recognize. Without a comprehensive understanding of semantics and morphology based on context, confusion errors can easily occur during recognition, resulting in low recognition accuracy.

[0003] Current mainstream methods for text image recognition in various languages ​​are not suitable for Lao text image recognition tasks. Furthermore, there are currently no publicly available annotated Lao text image datasets. This poses a challenge for deep learning research on Lao text image recognition, as it lacks sufficient Lao text image data for network training. To address the challenges faced in Lao text image recognition, innovative algorithms based on the unique glyph structure of the Lao language are needed, as well as efforts to construct Lao text image datasets to promote the application and development of deep learning techniques in this field. This will help improve the accuracy and robustness of Lao text image recognition technology, providing strong technical support for the digitization of Lao language and culture.

[0004] In response to the above problems, the present invention proposes a Laotian image text recognition method and device that integrates a language error correction model. Summary of the Invention

[0005] The present invention proposes a Laotian image text recognition method and device that integrates a language error correction model, which is used to solve the problem of poor recognition of similar characters in Laotian images and improve the accuracy and robustness of Laotian text image recognition technology.

[0006] The technical solution of the present invention is: a Laotian image text recognition method integrating a language error correction model, the method comprising:

[0007] Step 1: Construct a Lao text image dataset: Construct a Lao text image dataset through manual collection, annotation, and verification;

[0008] Step 2: Build a Laotian image text recognition model that integrates a language error correction model: Build a Transformer-based basic visual model, a language error correction module based on grammar correction, a feature shaping module based on the perceptron model, and a multi-dimensional feature fusion module based on cross-attention.

[0009] Design a text representation based on the Transformer-based visual model to extract Lao text image features and initial text image features;

[0010] A feature shaping module based on the perceptron model is designed to condense and reduce the dimension of the extracted text image features to obtain condensed image features.

[0011] Design a language error correction module based on grammatical correction, whose encoding layer encodes the text representation of the initial text image features recognized by the Transformer basic visual model to obtain the hidden layer representation of the source sequence;

[0012] A multi-dimensional feature fusion module based on cross-attention is designed to use syntactic rule constraints to decode the latent representation of the source sequence and the condensed image features, which is used to deeply mine the text information in Lao text images.

[0013] Step 3: Train the Laotian image-text recognition model based on the constructed fusion language error correction model;

[0014] Step 4. Lao text image recognition: Encapsulate the above trained converged model and deploy it on the server. Develop a set of API interfaces to call the Lao image text recognition model deployed on the server that integrates the language error correction model through the API to realize the recognition of Lao text images.

[0015] Furthermore, the specific steps of Step 1 are:

[0016] Step 1.1: We collected Lao text data from the internet and performed preprocessing operations such as deduplication, segmentation, and special character filtering to construct a Lao text label dataset for generating Lao text images. We also obtained Lao text images from real scenes through the Internet.

[0017] Step 1.2: Generate corresponding text images from the preprocessed Lao text labels using a text-image synthesis algorithm. The constructed image dataset covers Lao text images with various backgrounds, font colors, and styles, and is divided into training, validation, and test sets.

[0018] Step 1.3: Data diversity is improved through data enhancement methods. This step uses the following methods:

[0019] Scaling: Adjust the size of Lao text images to simulate different resolutions and scenarios;

[0020] Geometric transformation: geometric transformation of Lao text images by rotation, tilt and translation;

[0021] Occlusion and cropping: Randomly crop or center crop part of the Lao text image;

[0022] Noise: Add random noise to the Lao text image;

[0023] Lighting and contrast adjustment: Simulate the characteristics of text images in different lighting environments by adjusting brightness, contrast, and saturation.

[0024] Furthermore, in Step 2, a text representation of Lao text image features and initial text image features is designed based on the Transformer basic visual model, including:

[0025] Construct a basic visual feature extraction module based on Transformer, extract features from the input Lao text image through the visual Transformer, and convert the image into a token sequence; in the specific operation, first use the linear mapping layer to transform the original image X1∈R H1×W1×C1 Divide the image into P×P blocks, and then flatten these blocks into two-dimensional vectors Among them, P 2 Indicates the resolution of each small block, (P 2 C1) represents the number of channels, N1=H1×W1 / P 2 represents the length of the image block sequence, H1, W1, and C1 are the height, width, and number of channels of the image, respectively. Next, a linear mapping layer is used to map the image block to D dimensions, which serves as the input to the backbone network encoding layer. Similar to traditional visual Transformers, a D-dimensional learnable classification token is added to extract global information about the image. At the same time, each image block contains a learnable one-dimensional position embedding. The final encoder input is represented as:

[0026]

[0027] Among them, x cls ∈R 1×D Represents classification token, E∈R (N1+1)×D represents the linear mapping matrix, The vector representing the N1th image block, E pos ∈R (N1+1)×DRepresents position encoding; Z0 is input into the feature encoder for feature extraction. The feature encoder includes: a multi-head self-attention layer MSA for determining the relationship between feature vectors and a multi-layer perceptron MLP for performing feature extraction. The calculation formulas of the two are as follows:

[0028] Z′ m =MSA(LN(Z m-1 ))+Z m-1 ,m=1,…,12

[0029] Z m =MLP(LN(Z′ m ))+Z′ m ,m=1,…,12

[0030] Among them, the multi-layer perceptron consists of two linear activation layers, and the text image feature Z after the final feature encoding is m ∈R (11+1)×D Used for subsequent text recognition;

[0031] The text image feature Z m A series of linear projections are performed through the fully connected layer to obtain the text representation of the initial text image features.

[0032] Furthermore, in Step 2, a language error correction module based on grammatical correction is designed, whose encoding layer encodes the text representation of the initial text image features recognized by the Transformer basic visual model to obtain the hidden layer representation of the source sequence, including:

[0033] Construct a language error correction module based on grammatical correction. The basic process of the basic grammatical error correction model includes: first, the text representation X2 of the initial text image features containing grammatical errors is mapped through the encoder to obtain the output of the decoder. The specific formula of the decoder output is:

[0034] H=Encoder(X2,Θ e );

[0035] H is the hidden layer representation of the source sequence, Θ e are the learnable parameters of the encoder;

[0036] Then, the contextual relationship between words and sentence semantics are captured through the multi-head attention mechanism. The calculation formula is:

[0037]

[0038] Among them, Q is the query matrix, K is the parameter matrix, V is the specific value of the queried vector, d k It is a fixed value used to prevent the output from being too large;

[0039] The decoder dynamically generates the corrected text based on the hidden layer representation and the generated target sequence, and its formula is:

[0040] y t = Decoder(H, Y, Θ d );

[0041] where Y represents the generated partial sequence, and y t represents the next target prediction word, and Θ d is the learnable parameter of the encoder;

[0042] Up to this point, in the inference stage, the decoder generates the preliminary corrected target text according to the hidden layer representation H of the input source sequence through the greedy algorithm:

[0043]

[0044] where N represents the total number of time steps.

[0045] Furthermore, in Step 2, a feature shaping module based on the perceptron model is designed to凝练 and reduce the dimension of the extracted text image features to obtain the凝练 image features; including:

[0046] Construct a feature shaping module based on the perceptron model, which takes cross-attention as the core mechanism, where the initialized query matrix is Q ∈ W (N2×D1) , N2 represents the number of tokens in the current sequence, and D1 represents the channel dimension; at the same time, the high-dimensional matrix to be reduced in dimension is introduced as the key K ∈ W (M×D) and the value V ∈ W (M×C) in the attention formula. M is the number of tokens in the high-dimensional matrix, and C is the channel dimension. At this time, the product of the query Q and the key K is calculated according to the attention formula to obtain an N2*M matrix, and then further multiplied by the value V to obtain an N2*C low-dimensional matrix. At this time, since the number of tokens N2 in the query matrix << the number of tokens M in the high-dimensional matrix, a low-dimensional representation that can capture the main feature information of the high-dimensional matrix is finally obtained. To ensure the integrity of high-dimensional feature extraction, the low-dimensional matrix is iteratively updated through multiple self-attention and cross-attention calculations, continuously extracting the information of the high-dimensional matrix. The overall computational complexity is O(N2*M), but since N2 << M, the computational complexity can be effectively reduced to O(M), which is completely acceptable in practice. At the same time, to avoid losing the front and back semantic relationships of characters during the dimension reduction process, learnable position embeddings are used to label corresponding position tags for each character. After multiple rounds of iterative calculations, this module effectively凝练 the high-dimensional visual features and finally reduces them to a unified representation suitable for low-dimensional text feature interaction, thus laying a foundation for subsequent multi-modal tasks.

[0047] Furthermore, in Step 2, a cross-attention-based multi-dimensional feature fusion module is designed to perform decoding constraints on the latent representation of the source sequence and the condensed image features using a syntactic rule constraint strategy, so as to achieve in-depth mining of text information in Lao text images, including:

[0048] A multi-dimensional feature fusion module based on cross-attention is constructed. A new cross-attention layer is inserted between the encoding layer and the decoding layer of the grammatical correction-based language error correction module. The keys and values ​​of these attention layers are derived from the condensed image features generated by the feature shaping module, while the query comes from the hidden layer representation H of the output source sequence of the grammatical correction-based language error correction module. In the early stage of training the Laotian image text recognition model fused with the language error correction model, a tanh gating mechanism is introduced before the output of the multi-dimensional feature fusion module based on cross-attention enters the decoding layer. This mechanism performs a nonlinear transformation on the input through the hyperbolic tangent function tanh, compressing the data range to (-1, 1) to capture complex nonlinear relationships, while enhancing sensitivity to feature changes and suppressing the influence of extreme values. Specifically, the output of the multi-dimensional feature fusion module based on cross-attention is multiplied by the learnable scalar tanh(α) of the gating mechanism and then added to the input representation of the residual connection, where the scalar α is initialized to 0 to ensure that the initial model output is consistent with the pre-trained language error correction model.

[0049] The latent representation of the source sequence and the condensed image features are fused through a multi-dimensional feature fusion module based on cross-attention, and decoded through the decoding end of the language error correction module based on grammar correction to obtain the final output result.

[0050] Furthermore, the Step 3 includes:

[0051] During model training, the cross entropy loss function is used as the target optimization function of the Laotian image text recognition model that has been built with the fusion language error correction model. The calculation method is shown in the following formula:

[0052] L i =-∑lnP(Y t |M1,θ)(i=v,l)

[0053] Loss = λ v L v +λ l L l

[0054] Among them, M1 is the input Lao text image, θ is the model parameter of the current recognition network, and Y t |M1 is the label corresponding to the t-th feature sequence of the Lao text image, L v Represents the loss of the Transformer-based visual model, Ll represents the total loss of the language error correction module based on grammatical correction, the feature shaping module based on the perceptron model, and the multi-dimensional feature fusion module based on cross attention, λ v ,λ l represents the hyperparameter used to balance the loss.

[0055] Furthermore, the specific steps of Step 4 are:

[0056] Step 4.1. Implement batch processing of the Laotian image text recognition model. First, set the height of the model input image to 32 and the maximum width to W. max , for width less than W max The image is filled with pixels;

[0057] Step 4.2. Save the trained Lao text recognition model as a ".pth" file and load the model into memory through a web application based on the Sanic framework.

[0058] Step 4.3. Use the Sanic framework for back-end development, encapsulate the decoding process of the Lao text image text recognition model into a callable API interface, and complete the deployment to support multiple concurrent requests on the web port;

[0059] Step 4.4: Call the Lao text recognition model deployed on the server on the Web side, obtain the recognition results by inputting the Lao text image to be recognized, and display the recognition results on the front-end interface to complete the full process test.

[0060] The present invention provides a Laotian image text recognition device integrated with a language error correction model, comprising a module for executing the Laotian image text recognition method integrated with the language error correction model.

[0061] The beneficial effects of the present invention are:

[0062] 1. This method first uses a Transformer-based visual model to obtain Lao text image features. It then uses a feature shaping module based on a perceptron model to condense and reduce the extracted text image features to obtain condensed image features. It then uses the encoder layer of a grammatical correction-based language error correction module to encode the initial recognition results of the visual model to obtain error-corrected text features. Finally, a multi-dimensional feature fusion module based on cross-attention is used to fuse the condensed image features and error-corrected text features to obtain the final recognition results.

[0063] 2. This invention provides a Lao language error correction model for Lao text image recognition. By introducing grammatical knowledge, it further corrects errors that may occur in daily text image recognition. To ensure full utilization of visual feature information, a feature fusion module performs a secondary fusion of visual and language features, thereby enhancing the recognition ability of similar Lao characters.

[0064] 3. This paper uses a Transformer-based visual model to replace the traditional CNN structure. This structure can model the global information of the entire image by performing global attention calculation on visual features.

[0065] 4. Based on the original character sequence decoding, the present invention introduces a language error correction model, and performs secondary correction on the initial output of the visual model through grammatical knowledge to improve the semantic accuracy of the recognition result. At the same time, before the text features are input into the encoding layer of the language error correction model, a multi-dimensional feature fusion module based on cross-attention is designed to deeply fuse text and visual features to achieve information complementarity and synergistic enhancement between features. This fusion module effectively captures the potential correlation between text and visual features through the cross-attention mechanism, while improving the feature expression ability, it provides richer contextual information for the error correction model, and ultimately significantly improves the accuracy and robustness of Lao image text recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 This is a structural diagram of the feature shaping module based on the perceptron model in the present invention;

[0067] Figure 2 This is a model structure diagram of the Laotian image text recognition method integrated with the language error correction model in the present invention;

[0068] Figure 3 This is the overall flow chart of the Laotian image text recognition method that integrates the language error correction model in the present invention. DETAILED DESCRIPTION

[0069] Example 1: Figure 1-Figure 3 As shown, a Laotian image text recognition method integrating a language error correction model comprises:

[0070] Step 1. Construction of Lao text image dataset: To address the problem of lack of public Lao text image dataset in the current field, the present invention constructs a Lao text image dataset through manual collection, annotation and verification, providing data support for the Lao image text recognition method that integrates the language error correction model.

[0071] Furthermore, the specific steps of Step 1 are:

[0072] Step 1.1: We collected Lao text data from various fields on the internet and performed preprocessing operations such as deduplication, segmentation, and special character filtering to construct a Lao text label dataset for generating Lao text images. In addition, to enhance the diversity and authenticity of the dataset, we also obtained Lao text images in real scenes through the Internet.

[0073] Step 1.2: Using a text-image synthesis algorithm, the preprocessed Laotian text labels are converted into corresponding text images. To improve the robustness and generalization ability of the model, an image dataset was constructed that covers Laotian text images with various backgrounds, font colors, and styles, and is divided into training, validation, and test sets.

[0074] Step 1.3: To address the problem of insufficient Lao text images in real scenarios, we used data augmentation methods to increase data diversity. This step used the following methods:

[0075] Scaling: Adjust the size of Lao text images to simulate different resolutions and scenarios;

[0076] Geometric transformation: geometric transformation of Lao text images by rotation, tilt and translation;

[0077] Occlusion and cropping: Randomly crop or center crop part of the Lao text image;

[0078] Noise: Add random noise to the Lao text image;

[0079] Lighting and contrast adjustment: Simulate the characteristics of text images in different lighting environments by adjusting brightness, contrast, and saturation.

[0080] Furthermore, in Step 2, a text representation of Lao text image features and initial text image features is designed based on the Transformer basic visual model, including:

[0081] Construct a basic visual feature extraction module based on Transformer, extract features from the input Lao text image through the visual Transformer, and convert the image into a token sequence; this processing method enables the model to assign different attention levels to the entire image or each independent token, thereby mining deeper visual semantic information. In the specific operation, first, the original image X1∈R H1×W1×C1 Divide the image into P×P blocks, and then flatten these blocks into a two-dimensional vector X p ∈R N1×(P2C1) , where P 2 Indicates the resolution of each small block, (P 2C1) represents the number of channels, N1=H1×W1 / P 2 represents the length of the image block sequence, H1, W1, and C1 are the height, width, and number of channels of the image, respectively. Next, a linear mapping layer is used to map the image block to D dimensions, which serves as the input to the backbone network encoding layer. Similar to traditional visual Transformers, a D-dimensional learnable classification token is added to extract global information about the image. At the same time, each image block contains a learnable one-dimensional position embedding. The final encoder input is represented as:

[0082]

[0083] Among them, x cls ∈R 1×D Represents classification token, E∈R (N1+1)×D represents the linear mapping matrix, The vector representing the N1th image block, E pos ∈R (N1+1)×D Represents position encoding; Z0 is input into the feature encoder for feature extraction. The feature encoder includes: a multi-head self-attention layer MSA for determining the relationship between feature vectors and a multi-layer perceptron MLP for performing feature extraction. The calculation formulas of the two are as follows:

[0084] Z′ m =MSA(LN(Z m-1 ))+Z m-1 ,m=1,…,12

[0085] Z m =MLP(LN(Z′ m ))+Z′ m ,m=1,…,12

[0086] Among them, the multi-layer perceptron consists of two linear activation layers, and the text image feature Z after the final feature encoding is m ∈R (N1+1)×D Used for subsequent text recognition;

[0087] The text image feature Z m A series of linear projections are performed through the fully connected layer to obtain the text representation of the initial text image features.

[0088] Step 2: Build a Laotian image text recognition model that integrates a language error correction model: Build a Transformer-based basic visual model, a language error correction module based on grammar correction, a feature shaping module based on the perceptron model, and a multi-dimensional feature fusion module based on cross-attention.

[0089] Design a text representation based on the Transformer-based visual model to extract Lao text image features and initial text image features;

[0090] A feature shaping module based on the perceptron model is designed to condense and reduce the dimension of the extracted text image features to obtain condensed image features.

[0091] Design a language error correction module based on grammatical correction, whose encoding layer encodes the text representation of the initial text image features recognized by the Transformer basic visual model to obtain the hidden layer representation of the source sequence;

[0092] A multi-dimensional feature fusion module based on cross-attention was designed to apply syntactic rule constraints to the latent representation of the source sequence and the condensed image features, enabling in-depth mining of textual information in Laotian text images. This process significantly enhanced the model's recognition accuracy for similar characters in Laotian text images.

[0093] Furthermore, in Step 2, a language error correction module based on grammatical correction is designed, whose encoding layer encodes the text representation of the initial text image features recognized by the Transformer basic visual model to obtain the hidden layer representation of the source sequence, including:

[0094] Construct a language error correction module based on grammatical correction. The basic process of the basic grammatical error correction model includes: first, the text representation X2 of the initial text image features containing grammatical errors is mapped through the encoder to obtain the output of the decoder. The specific formula of the decoder output is:

[0095] H=Encoder(X2,Θ e );

[0096] H is the hidden layer representation of the source sequence, Θ e are the learnable parameters of the encoder;

[0097] Then, the contextual relationship between words and sentence semantics are captured through the multi-head attention mechanism. The calculation formula is:

[0098]

[0099] Among them, Q is the query matrix, K is the parameter matrix, V is the specific value of the queried vector, d k It is a fixed value used to prevent the output from being too large;

[0100] The decoder dynamically generates the corrected text through the hidden layer representation and the generated target sequence. The formula is:

[0101] y t =Decoder(H,Y,Θd )

[0102] where Y represents the generated partial sequence, and y t represents the next target prediction word, and Θ d are the learnable parameters of the encoder;

[0103] Up to this point, in the inference stage, the decoder generates a preliminary corrected target text according to the hidden layer representation H of the input source sequence through the greedy algorithm:

[0104]

[0105] where N represents the total number of time steps.

[0106] Furthermore, in Step 2, a feature shaping module based on the perceptron model is designed to refine and reduce the dimension of the extracted text image features to obtain refined image features, including:

[0107] Construct a feature shaping module based on the perceptron model. The perceptron is an architecture based on the attention mechanism that can reduce the dimension of high-dimensional inputs such as images and videos without requiring separate adaptation for a specific domain. This module takes cross-attention as the core mechanism, where the initialized query matrix is Q ∈ W (N2×D1) , N2 represents the number of tokens in the current sequence, and D1 represents the channel dimension; at the same time, the high-dimensional matrix to be reduced in dimension is used as the key K ∈ W (M×D) and the value V ∈ W (M×C) are introduced, M is the number of tokens in the high-dimensional matrix, C is the channel dimension. At this time, the product of the query Q and the key K is calculated according to the attention formula to obtain a matrix of N2 * M, and then further multiplied by the value V to obtain a low-dimensional matrix of N2 * C. At this time, since the number of tokens N2 in the query matrix << the number of tokens M in the high-dimensional matrix, a low-dimensional representation that can capture the main feature information of the high-dimensional matrix is finally obtained. To ensure the integrity of high-dimensional feature extraction, the low-dimensional matrix is iteratively updated through multiple self-attention and cross-attention calculations, continuously extracting the information of the high-dimensional matrix. The overall computational complexity is O(N2 * M), but since N2 << M, the computational complexity can be effectively reduced to O(M), which is completely acceptable in practice. At the same time, to avoid losing the semantic relationship between characters before and after during the dimension reduction process, learnable position embeddings are used to label corresponding position tags for each character. After multiple rounds of iterative calculations, this module effectively refines the high-dimensional visual features and finally reduces them to a unified representation suitable for low-dimensional text feature interaction, thus laying a foundation for subsequent multi-modal tasks.

[0108] Furthermore, in Step 2, a cross-attention-based multi-dimensional feature fusion module is designed to perform decoding constraints on the latent representation of the source sequence and the condensed image features using a syntactic rule constraint strategy, so as to achieve in-depth mining of text information in Lao text images, including:

[0109] A cross-attention based multi-dimensional feature fusion module is constructed to fully utilize the visual features extracted by the visual encoder while dynamically adjusting the behavior of the language error correction model. A new cross-attention layer is inserted between the encoding layer and the decoding layer of the grammatical correction-based language correction module. The keys and values ​​of these attention layers are derived from the condensed image features generated by the feature shaping module, while the query comes from the hidden layer representation H of the output source sequence of the grammatical correction-based language correction module. To prevent the fusion module from interfering with the original language correction model, a tanh gating mechanism is introduced before the output of the multi-dimensional feature fusion module based on cross-attention enters the decoding layer in the early stage of training the Laotian image text recognition model fused with the language correction model. This mechanism performs a nonlinear transformation on the input using the hyperbolic tangent function tanh, compressing the data range to (-1, 1) to capture complex nonlinear relationships, while enhancing sensitivity to feature changes and suppressing the influence of extreme values. Specifically, the output of the multi-dimensional feature fusion module based on cross-attention is multiplied by the learnable scalar tanh(α) of the gating mechanism and then added to the input representation of the residual connection. The scalar α is initialized to 0 to ensure that the initial model output is consistent with the pre-trained language correction model, thereby improving training stability and final performance.

[0110] The latent representation of the source sequence and the condensed image features are fused through a multi-dimensional feature fusion module based on cross-attention, and decoded through the decoding end of the language error correction module based on grammar correction to obtain the final output result.

[0111] Step 3: Train the Laotian image-text recognition model based on the constructed fusion language error correction model;

[0112] Furthermore, the Step 3 includes:

[0113] During model training, the cross entropy loss function is used as the target optimization function of the Laotian image text recognition model that has been built with the fusion language error correction model. The calculation method is shown in the following formula:

[0114] L i =-∑lnP(Y t |M1,θ)(i=v,l)

[0115] Loss = λ v L v +λ l L l

[0116] Among them, M1 is the input Lao text image, θ is the model parameter of the current recognition network, and Y t |M1 is the label corresponding to the t-th feature sequence of the Lao text image, L v Represents the loss of the Transformer-based visual model, L l represents the total loss of the language error correction module based on grammatical correction, the feature shaping module based on the perceptron model, and the multi-dimensional feature fusion module based on cross attention, λ v ,λ l represents the hyperparameter used to balance the loss, and is simply set to 1 in our experiments.

[0117] Step 4, Lao Text Image Recognition: The trained and converged model was packaged and deployed on the server. An API was developed to call the Lao text image recognition model deployed on the server, which was integrated with the language error correction model. This API was used to recognize Lao text images. To improve the efficiency of Lao text image recognition, the relevant model input parameters were optimized, batch processing of image data was implemented, and the trained and converged model was packaged for deployment on the server. Based on this, a set of APIs was developed to achieve fast and efficient recognition of Lao text images.

[0118] Furthermore, the specific steps of Step 4 are:

[0119] Step 4.1. Implement batch processing of the Laotian image text recognition model. First, set the height of the model input image to 32 and the maximum width to W. max , for width less than W max The image is filled with pixels;

[0120] Step 4.2. Save the trained Lao text recognition model as a ".pth" file and load the model into memory through a web application based on the Sanic framework.

[0121] Step 4.3. Use the Sanic framework for back-end development, encapsulate the decoding process of the Lao text image text recognition model into a callable API interface, and complete the deployment to support multiple concurrent requests on the web port;

[0122] Step 4.4: Call the Lao text recognition model deployed on the server on the Web side, obtain the recognition results by inputting the Lao text image to be recognized, and display the recognition results on the front-end interface to complete the full process test.

[0123] The present invention provides a Laotian image and text recognition device that integrates a language error correction model, including a module for executing the Laotian image and text recognition method that integrates a language error correction model. Specifically, the device may include:

[0124] Lao text image dataset construction module: used to construct a Lao text image dataset through manual collection, annotation and verification;

[0125] Laotian image-text recognition model construction module integrated with language error correction model: used to build a basic visual model based on Transformer, a language error correction module based on grammar correction, a feature shaping module based on the perceptron model, and a multi-dimensional feature fusion module based on cross-attention;

[0126] Design a text representation based on the Transformer-based visual model to extract Lao text image features and initial text image features;

[0127] A feature shaping module based on the perceptron model is designed to condense and reduce the dimension of the extracted text image features to obtain condensed image features.

[0128] Design a language error correction module based on grammatical correction, whose encoding layer encodes the text representation of the initial text image features recognized by the Transformer basic visual model to obtain the hidden layer representation of the source sequence;

[0129] A multi-dimensional feature fusion module based on cross-attention is designed to use syntactic rule constraints to decode the latent representation of the source sequence and the condensed image features, which is used to deeply mine the text information in Lao text images.

[0130] Training module: used to train the Lao image text recognition model that integrates the constructed language error correction model;

[0131] Lao text image recognition module: It is used to encapsulate the above-mentioned trained converged model and deploy it on the server. It also develops a set of API interfaces and calls the Lao image text recognition model integrated with the language error correction model deployed on the server through the API to realize the recognition of Lao text images.

[0132] To validate the effectiveness of the proposed Laotian image-to-text recognition model based on multi-granularity decoding constraints, experiments were conducted on 5 million self-constructed Laotian text images. Approximately 30,000 of these images were manually annotated, and the remaining data was algorithmically synthesized. To ensure fairness, we used uniform experimental conditions for all comparative experiments: the input image size was 32×128, the Adam optimizer was used, and the initial learning rate was set to 1. The CosineAnnealing strategy was used to gradually reduce the learning rate to ensure a smaller learning rate when the model was most likely to find a solution. The batch size was set to 128, and training was performed for 10 epochs.

[0133] The following comparative experiment was designed for analysis.

[0134] The experiment uses Lao Sequence Accuracy (SA) as the evaluation indicator, as shown below:

[0135]

[0136] Among them, SA, SL, and LN represent the sequence accuracy, the total number of correct sequences, and the total number of sequences of Lao text image recognition, respectively.

[0137] Experiment 1: Main comparative experimental results;

[0138] Table 1 Main experimental results

[0139]

[0140]

[0141] As shown in Table 1, the proposed recognition model achieved an accuracy of 95.3% in the Lao text recognition task, achieving state-of-the-art performance. This accuracy improved by 8.2% compared to the connectionist temporal classification (CTC) method and by 10.4% compared to the sequence-to-sequence (Seq2Seq) method. This result demonstrates that the proposed method effectively enhances the ability to extract Lao image features by introducing a visual Transformer encoding module for global context modeling. Furthermore, compared to Transformer-based recognition methods, the accuracy improved by 3.6%, demonstrating that the language error correction model can effectively distinguish similar-shaped Lao text characters during recognition, optimizing the model's recognition performance. Furthermore, compared to a language model-based iterative recognition method, the accuracy improved by 1.8%. This further demonstrates that, through its multidimensional feature fusion mechanism, the proposed method not only maintains character-level recognition accuracy but also fully exploits the potential information in visual features, correcting the language model encoder and significantly improving overall recognition accuracy and robustness.

[0142] Experiment 2: ablation experiment results;

[0143] To verify the effectiveness of the feature shaping module and the change in feature fusion position, we conducted an ablation experiment on the baseline model. The experimental results are shown in Table 2 ("×" indicates not used, "√" indicates used):

[0144] Table 2 shows the influence of feature shaping module and feature fusion position on recognition

[0145]

[0146] As shown in Table 2, the experimental results show that introducing the feature shaping module can effectively improve the performance of the fusion module, thereby improving the accuracy of model recognition. At the same time, changing the feature fusion position can also effectively improve the utilization of visual features and enhance the main performance of the model. This proves the effectiveness of the proposed method.

[0147] Experiment 3: Experimental results on the real test set;

[0148] In order to verify the performance of the invented model in training and inference speed, experiments were conducted on a dataset of 1,000 real-scene Lao text images.

[0149] Table 3 shows the experimental results on the real test set.

[0150]

[0151] As shown in Table 3, our method achieved a recognition accuracy of 93.2% in real-world scenarios, significantly outperforming other comparison methods. The accuracy of the ResNet50+BiLSTM+CTC method was 88.6%, the ResNet50+Transformer method was 90.8%, and the ABINet method was 92.4%. Our method's advantages in multidimensional feature fusion and global context modeling enabled it to achieve the highest recognition accuracy in Lao text recognition tasks, further validating its robustness and effectiveness in complex scenarios.

[0152] The specific embodiments of the present invention are described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Various changes can be made within the knowledge of ordinary technicians in this field without departing from the scope of the present invention.

Claims

1. A Laotian image text recognition method integrating a language error correction model is characterized by: The method comprises: Step 1: Construct a Lao text image dataset: Construct a Lao text image dataset through manual collection, annotation, and verification; Step 2: Build a Laotian image text recognition model that integrates a language error correction model: Build a Transformer-based basic visual model, a language error correction module based on grammar correction, a feature shaping module based on the perceptron model, and a multi-dimensional feature fusion module based on cross-attention. Design a text representation based on the Transformer-based visual model to extract Lao text image features and initial text image features; A feature shaping module based on the perceptron model is designed to condense and reduce the dimension of the extracted text image features to obtain condensed image features. Design a language error correction module based on grammatical correction, whose encoding layer encodes the text representation of the initial text image features recognized by the Transformer basic visual model to obtain the hidden layer representation of the source sequence; A multi-dimensional feature fusion module based on cross-attention is designed to use syntactic rule constraints to decode the latent representation of the source sequence and the condensed image features, which is used to deeply mine the text information in Lao text images. include: A multi-dimensional feature fusion module based on cross-attention is constructed. A new cross-attention layer is inserted between the encoding layer and the decoding layer of the grammatical correction-based language error correction module. The keys and values ​​of these attention layers are derived from the condensed image features generated by the feature shaping module, while the query comes from the hidden layer representation H of the output source sequence of the grammatical correction-based language error correction module. In the early stage of training the Laotian image text recognition model fused with the language error correction model, a tanh gating mechanism is introduced before the output of the multi-dimensional feature fusion module based on cross-attention enters the decoding layer. This mechanism performs a nonlinear transformation on the input through the hyperbolic tangent function tanh, compressing the data range to (-1, 1) to capture complex nonlinear relationships, while enhancing sensitivity to feature changes and suppressing the influence of extreme values. Specifically, the output of the multi-dimensional feature fusion module based on cross-attention is multiplied by the learnable scalar tanh(α) of the gating mechanism and then added to the input representation of the residual connection, where the scalar α is initialized to 0 to ensure that the initial model output is consistent with the pre-trained language error correction model. The latent representation of the source sequence and the condensed image features are fused through a multi-dimensional feature fusion module based on cross attention, and decoded through the decoding end of the language error correction module based on grammar correction to obtain the final output result; Step 3: Train the Laotian image-text recognition model based on the constructed fusion language error correction model; Step 4. Lao text image recognition: Encapsulate the trained converged model and deploy it on the server. Develop an API interface to call the Lao image text recognition model deployed on the server that integrates the language error correction model through the API to realize the recognition of Lao text images.

2. The Laotian image text recognition method integrating the language error correction model according to claim 1 is characterized in that: The specific steps of Step 1 are: Step 1.1: We collected Lao text data from the internet and performed preprocessing operations such as deduplication, segmentation, and special character filtering to construct a Lao text label dataset for generating Lao text images. We also obtained Lao text images from real scenes through the Internet. Step 1.2: Generate corresponding text images from the preprocessed Lao text labels using a text-image synthesis algorithm. The constructed image dataset covers Lao text images with various backgrounds, font colors, and styles, and is divided into training, validation, and test sets. Step 1.3: Data diversity is improved through data enhancement methods. This step uses the following methods: Scaling: Adjust the size of Lao text images to simulate different resolutions and scenarios; Geometric transformation: geometric transformation of Lao text images by rotation, tilt and translation; Occlusion and cropping: Randomly crop or center crop part of the Lao text image; Noise: Add random noise to the Lao text image; Lighting and contrast adjustment: Simulate the characteristics of text images in different lighting environments by adjusting brightness, contrast, and saturation.

3. The Laotian image-text recognition method integrating a language error correction model according to claim 1 is characterized in that: In Step 2, a text representation of Lao text image features and initial text image features is designed based on the Transformer basic visual model, including: Construct a basic visual feature extraction module based on Transformer, extract features from the input Lao text image through the visual Transformer, and convert the image into a token sequence; in the specific operation, first use the linear mapping layer to transform the original image X1∈R H1×W1×C1 Divide the image into P×P blocks, and then flatten these blocks into two-dimensional vectors Among them, P 2 Indicates the resolution of each small block, N1=H1×W1 / P 2 represents the length of the image block sequence, H1, W1, and C1 are the height, width, and number of channels of the image, respectively. Next, a linear mapping layer is used to map the image block to D dimensions, which serves as the input to the backbone network encoding layer. Similar to traditional visual Transformers, a D-dimensional learnable classification token is added to extract global information about the image. At the same time, each image block contains a learnable one-dimensional position embedding. The final encoder input is represented as: Among them, x cls ∈R 1×D Represents classification token, E∈R (N1+1)×D represents the linear mapping matrix, The vector representing the N1th image block, E pos ∈R (N1+1)×D Represents position encoding; Z0 is input into the feature encoder for feature extraction. The feature encoder includes: a multi-head self-attention layer MSA for determining the relationship between feature vectors and a multi-layer perceptron MLP for performing feature extraction. The calculation formulas of the two are as follows: Z′ m =MSA(LN(Z m-1 ))+Z m-1 ,m=1,…,12 WITH m =MLP(LN(Z′ m ))+Z′ m ,m=1,…,12 Among them, the multi-layer perceptron consists of two linear activation layers, and the text image feature Z after the final feature encoding is m ∈R (N1 +1)×D Used for subsequent text recognition; The text image feature Z m A series of linear projections are performed through the fully connected layer to obtain the text representation of the initial text image features.

4. The Laotian image-text recognition method integrating a language error correction model according to claim 1 is characterized in that: In Step 2, a language error correction module based on grammatical correction is designed. Its encoding layer encodes the text representation of the initial text image features recognized by the Transformer basic visual model to obtain the hidden layer representation of the source sequence, including: Construct a language error correction module based on grammatical correction. The basic process of the basic grammatical error correction model includes: first, the text representation X2 of the initial text image features containing grammatical errors is mapped through the encoder to obtain the output of the decoder. The specific formula of the decoder output is: H=Encoder(X2,Θ e ); H is the hidden layer representation of the source sequence, Θ e are the learnable parameters of the encoder; Then, the contextual relationship between words and sentence semantics are captured through the multi-head attention mechanism. The calculation formula is: Among them, Q is the query matrix, K is the parameter matrix, V is the specific value of the queried vector, d k It is a fixed value used to prevent the output from being too large; The decoder dynamically generates the corrected text through the hidden layer representation and the generated target sequence. The formula is: y t =Decoder(H,Y,Θ d ); Among them, Y represents the generated partial sequence, y t represents the next target prediction word, Θ d are the learnable parameters of the encoder; At this point, in the inference phase, the decoder generates a preliminary corrected target text through a greedy algorithm based on the hidden layer representation H of the input source sequence: Where N represents the total number of time steps.

5. The Laotian image-text recognition method integrating a language error correction model according to claim 1 is characterized in that: In Step 2, a feature shaping module based on a perceptron model is designed to condense and reduce the dimension of the extracted text image features to obtain condensed image features; including: Construct a feature shaping module based on the perceptron model, which uses cross attention as the core mechanism, where the initialized query matrix is ​​Q∈W (N2×D1) , N2 represents the number of tokens in the current sequence, D1 represents the channel dimension; at the same time, the high-dimensional matrix to be reduced is used as the key K∈W in the attention formula (M×D) Sum value V∈W (M×C) In this example, M is the number of tokens in the high-dimensional matrix, and C is the channel dimension. The product of the query Q and the key K is calculated according to the attention formula to obtain an N2*M matrix, which is further multiplied by the value V to obtain an N2*C low-dimensional matrix. D represents the channel dimension.

6. The Laotian image text recognition method integrating a language error correction model according to claim 1 is characterized in that: Step 3 includes: During model training, the cross entropy loss function is used as the target optimization function of the Laotian image text recognition model that has been built with the fusion language error correction model. The calculation method is shown in the following formula: Loss=λ v L v +λ l L l Among them, M1 is the input Lao text image, θ is the model parameter of the current recognition network, and Y t |M1 is the label corresponding to the t-th feature sequence of the Lao text image, L v Represents the loss of the Transformer-based visual model, L l represents the total loss of the language error correction module based on grammatical correction, the feature shaping module based on the perceptron model, and the multi-dimensional feature fusion module based on cross attention, λ v ,λ l represents the hyperparameter used to balance the loss.

7. The Laotian image text recognition method integrating a language error correction model according to claim 1 is characterized in that: The specific steps of Step 4 are: Step 4.

1. Implement batch processing of the Laotian image text recognition model. First, set the height of the model input image to 32 and the maximum width to W. max , for width less than W max The image is filled with pixels; Step 4.

2. Save the trained Lao text recognition model as a ".pth" file and load the model into memory through a web application based on the Sanic framework. Step 4.

3. Use the Sanic framework for back-end development, encapsulate the decoding process of the Lao text image text recognition model into a callable API interface, and complete the deployment to support multiple concurrent requests on the web port; Step 4.4: Call the Lao text recognition model deployed on the server on the Web side, obtain the recognition results by inputting the Lao text image to be recognized, and display the recognition results on the front-end interface to complete the full process test.

8. The Laotian image text recognition device integrating the language error correction model is characterized by: The method comprises a module for executing the Lao image text recognition method integrating the language error correction model as described in any one of claims 1 to 7.

Images