Lao image text recognition method and device fused with language error correction model

Abstract

The invention relates to a Lao image text recognition method and device fused with a language error correction model, and belongs to the field of natural language processing. Lao character structures are more complex and have numerous homomorphic words, and the method is provided in order to solve the problem that similar characters in Lao images are poor in recognition. The Lao text image recognition method comprises the steps of Lao text image data set construction, Lao image text recognition model construction fused with a language error correction model, Lao image text recognition model training fused with the language error correction model and Lao text image recognition. The Lao image text recognition device fused with the language error correction model is manufactured according to functional modularization of the four parts, and the problem that Lao characters are difficult to recognize due to similar font structures is effectively solved.

Description

Technical Field

[0001] The invention provides a Laotian image text recognition method and device integrating a language error correction model, belonging to the technical field of natural language processing. Background Art

[0002] Lao has a unique coding order and character combination rules. The character structure of Lao is more complex. Lao uses a combination of vowels, consonants, and tone symbols in an up-down, left-right structure. The relative positions between characters are complex, far exceeding the linear arrangement of English. In addition, Lao characters often have no clear spacing, and continuous writing or overlapping phenomena are common, which leads to problems with character segmentation and recognition. In addition, there are many homographs in Lao, and individual characters are difficult to accurately recognize. If the semantics and morphology are not comprehensively judged based on the context, it is easy to make confusion errors during the recognition process, resulting in low recognition accuracy.

[0003] The current mainstream text image recognition methods for major languages ​​are not applicable to Lao text image recognition tasks. In addition, there is no publicly annotated Lao text image dataset at this stage, which means that when using deep learning technology to study Lao text image recognition technology, there is a lack of a large amount of Lao text image data available for network training. In order to solve the challenges faced in the field of Lao text image recognition, it is necessary to design innovative algorithms based on the special glyph structure of Lao itself, and also to work hard to build a Lao text image dataset to promote the application and development of deep learning technology in this field. This will help improve the accuracy and robustness of Lao text image recognition technology, and provide firm technical support for promoting the digitization of Lao language and culture.

[0004] In view of the above problems, the present invention proposes a Lao image text recognition method and device integrating a language error correction model. Summary of the invention

[0005] The present invention proposes a Lao image text recognition method and device integrating a language error correction model, which are used to solve the problem of poor recognition of similar characters in Lao images and improve the accuracy and robustness of Lao text image recognition technology.

[0006] The technical solution of the present invention is: a Laotian image text recognition method integrating a language error correction model, the method comprising:

[0007] Step 1: Construction of Lao text image dataset: A Lao text image dataset is constructed by manual collection, annotation and verification;

[0008] Step 2: Construction of Laotian image text recognition model integrating language error correction model: Construct a language error correction module based on Transformer basic visual model, a feature shaping module based on perceptron model, and a multi-dimensional feature fusion module based on cross attention;

[0009] Design a text representation based on the Transformer-based visual model to extract Lao text image features and initial text image features;

[0010] Design a feature shaping module based on the perceptron model to condense and reduce the dimension of the extracted text image features to obtain condensed image features;

[0011] Design a language error correction module based on grammar correction, whose encoding layer encodes the text representation of the initial text image features recognized by the Transformer basic visual model to obtain the hidden layer representation of the source sequence;

[0012] A multi-dimensional feature fusion module based on cross attention is designed to use syntactic rule constraint strategy to decode the latent representation and condensed image features of the source sequence, so as to realize the deep mining of text information in Lao text images.

[0013] Step 3: Train the Lao image text recognition model that has been built to integrate the language error correction model;

[0014] Step 4, Lao text image recognition: Encapsulate the above trained converged model and deploy it on the server, develop a set of API interfaces, and call the Lao image text recognition model integrated with the language error correction model deployed on the server through the API to realize the recognition of Lao text images.

[0015] Furthermore, the specific steps of Step 1 are:

[0016] Step 1.1, collect Lao text data from the Internet, perform deduplication, segmentation and special character filtering preprocessing operations, and build a set of Lao text label data sets for generating Lao text images; also obtain Lao text images in real scenes through the Internet;

[0017] Step 1.2, generate corresponding text images from the preprocessed Lao text labels through a text-image synthesis algorithm; the constructed image dataset covers Lao text images of various backgrounds, font colors and different styles, and is divided into training set, validation set and test set;

[0018] Step 1.3: Data enhancement methods are used to improve data diversity. This step uses the following methods:

[0019] Scaling: Adjust the size of Lao text images to simulate different resolutions and scenarios;

[0020] Geometric transformation: geometric transformation of Lao text images by rotation, tilt and translation;

[0021] Occlusion and cropping: Randomly crop or center crop part of the Lao text image;

[0022] Noise: Add random noise to the Lao text image;

[0023] Lighting and contrast adjustment: Simulate the text image characteristics under different lighting environments by adjusting brightness, contrast and saturation.

[0024] Furthermore, in the Step 2, a text representation of Lao text image features and initial text image features is designed based on the Transformer basic visual model, including:

[0025] Construct a basic visual feature extraction module based on Transformer, extract features from the input Lao text image through the visual Transformer, and convert the image into a token sequence; in the specific operation, first use the linear mapping layer to transform the original image X1∈R H1×W1×C1 Divide into P×P blocks, and then flatten these image blocks into two-dimensional vectors Where P 2 Indicates the resolution of each small block, (P 2 C1) represents the number of channels, N1=H1×W1 / P 2 Represents the length of the image block sequence, H1, W1, and C1 are the height, width, and number of channels of the image, respectively; then, a linear mapping layer is used to map the image block to D dimensions as the input of the backbone network encoding layer; similar to the traditional visual Transformer, a D-dimensional learnable classification token is added to extract the global information of the image; at the same time, each image block contains a learnable one-dimensional position embedding, and the final encoder input is expressed as:

[0026]

[0027] Among them, x cls ∈R 1×D Represents classification token, E∈R (N1+1)×D represents the linear mapping matrix, The vector representing the N1th image block, E pos ∈R (N1+1)×DRepresents position encoding; Z0 is input into the feature encoder for feature extraction. The feature encoder includes: the multi-head self-attention layer MSA is used to determine the relationship between feature vectors, and the multi-layer perceptron MLP is used to perform feature extraction. The calculation formulas of the two are as follows:

[0028] Z′ m =MSA(LN(Z m-1 ))+Z m-1 ,m=1,…,12

[0029] Z m =MLP(LN(Z′ m ))+Z′ m ,m=1,…,12

[0030] Among them, the multi-layer perceptron consists of two layers of linear activation layers, and the final feature-encoded text image feature Z m ∈R (11+1)×D Used for subsequent text recognition;

[0031] The text image feature Z m A series of linear projections are performed through the fully connected layer to obtain the text representation of the initial text image features.

[0032] Furthermore, in the Step 2, a language error correction module based on grammar correction is designed, and its encoding layer encodes the text representation of the initial text image features recognized based on the Transformer basic visual model to obtain the hidden layer representation of the source sequence, including:

[0033] Construct a language error correction module based on grammatical correction. The basic process of the basic grammatical error correction model includes: first, the text representation X2 of the initial text image features containing grammatical errors is mapped through the encoder to obtain the output of the decoder. The specific formula of the decoder output is:

[0034] H = Encoder(X2, Θ e );

[0035] H is the hidden layer representation of the source sequence, Θ e are the learnable parameters of the encoder;

[0036] Then, the contextual relationship between words and sentence semantics are captured through the multi-head attention mechanism. The calculation formula is:

[0037]

[0038] Among them, Q is the query matrix, K is the parameter matrix, V is the specific value of the queried vector, and d k It is a fixed value used to prevent the output result from being too large;

[0039] The decoder dynamically generates the corrected text based on the hidden layer representation and the generated target sequence, and its formula is:

[0040] y t = Decoder(H, Y, Θ d );

[0041] where Y represents the generated partial sequence, y t represents the next target prediction word, and Θ d is the learnable parameter of the encoder;

[0042] So far, in the inference stage, the decoder generates the preliminary corrected target text according to the hidden layer representation H of the input source sequence through the greedy algorithm:

[0043]

[0044] where N represents the total number of time steps.

[0045] Furthermore, in Step 2, a feature shaping module based on the perceptron model is designed to凝练 and reduce the dimension of the extracted text image features to obtain the凝练 image features; including:

[0046] Construct a feature shaping module based on the perceptron model, which takes cross-attention as the core mechanism, where the initialized query matrix is Q ∈ W (N2×D1) , N2 represents the number of tokens in the current sequence, and D1 represents the channel dimension; at the same time, the high-dimensional matrix to be reduced in dimension is introduced as the key K ∈ W (M×D) and the value V ∈ W (M×C) in the attention formula. M is the number of tokens in the high-dimensional matrix, and C is the channel dimension. At this time, the product of the query Q and the key K is calculated according to the attention formula to obtain an N2 * M matrix, and then further multiplied by the value V to obtain an N2 * C low-dimensional matrix. At this time, since the number of tokens N2 in the query matrix << the number of tokens M in the high-dimensional matrix, a low-dimensional representation that can capture the main feature information of the high-dimensional matrix is finally obtained. To ensure the integrity of high-dimensional feature extraction, the low-dimensional matrix is iteratively updated through multiple self-attention and cross-attention calculations, continuously extracting the information of the high-dimensional matrix. The overall computational complexity is O(N2 * M), but since N2 << M, the computational complexity can be effectively reduced to O(M), which is completely acceptable in practice. At the same time, to avoid losing the semantic relationship between characters before and after during the dimension reduction process, learnable position embeddings are used to label corresponding position tags for each character. After multiple rounds of iterative calculations, this module effectively凝练 the high-dimensional visual features and finally reduces them to a unified representation suitable for low-dimensional text feature interaction, thus laying a foundation for subsequent multi-modal tasks.

[0047] Furthermore, in the Step 2, a multi-dimensional feature fusion module based on cross attention is designed to perform decoding constraints on the latent representation of the source sequence and the condensed image features using a syntactic rule constraint strategy, so as to realize the deep mining of text information in Lao text images, including:

[0048] A multi-dimensional feature fusion module based on cross-attention is constructed. A new cross-attention layer is inserted between the encoding layer and the decoding layer of the language error correction module based on grammar correction. The keys and values ​​of these attention layers are derived from the condensed image features generated by the feature shaping module, while the query comes from the hidden layer representation H of the output source sequence of the language error correction module based on grammar correction. In the early stage of training the Lao image text recognition model fused with the language error correction model, a tanh gating mechanism is introduced before the output of the multi-dimensional feature fusion module based on cross-attention enters the decoding layer. This mechanism performs a nonlinear transformation on the input through the hyperbolic tangent function tanh, compressing the data range to (-1,1) to capture complex nonlinear relationships, while enhancing the sensitivity to feature changes and suppressing the influence of extreme values. Specifically, the output of the multi-dimensional feature fusion module based on cross-attention is multiplied by the learnable scalar tanh(α) of the gating mechanism and then added to the input representation of the residual connection, where the scalar α is initialized to 0 to ensure that the initial model output is consistent with the pre-trained language error correction model.

[0049] The latent representation of the source sequence and the condensed image features are fused through a multi-dimensional feature fusion module based on cross-attention, and decoded through the decoding end of the language error correction module based on grammar correction to obtain the final output result.

[0050] Furthermore, the Step 3 includes:

[0051] When training the model, the cross entropy loss function is used as the target optimization function of the Lao image text recognition model that has been built with the fusion language error correction model. The calculation method is shown in the following formula:

[0052] L i =-∑lnP(Y t |M1,θ)(i=v,l)

[0053] Loss = λ v L v +λ l L l

[0054] Among them, M1 is the input Lao text image, θ is the model parameter of the current recognition network, and Y t |M1 is the label corresponding to the tth feature sequence of the Lao text image, L v Represents the loss of the Transformer-based visual model, Ll represents the total loss of the language error correction module based on grammar correction, the feature shaping module based on the perceptron model, and the multi-dimensional feature fusion module based on cross attention, λ v , l represents the hyperparameter used to balance the loss.

[0055] Furthermore, the specific steps of Step 4 are:

[0056] Step 4.1: Implement batch processing of the Lao image text recognition model. First, set the height of the model input image to 32 and the maximum width to W. max , for width less than W max The image is filled by pixel padding;

[0057] Step 4.2, save the trained Lao text recognition model as a ".pth" format file, and load the model into memory through a web application based on the Sanic framework;

[0058] Step 4.3. Use the Sanic framework for backend development, encapsulate the decoding process of the Lao text image text recognition model into a callable API interface, and complete the deployment to support multiple concurrent requests of the Web port;

[0059] Step 4.4: Call the Lao text recognition model deployed on the server on the Web side, obtain the recognition result by inputting the Lao text image to be recognized, and display the recognition result on the front-end interface to complete the full process test.

[0060] The invention provides a Lao image text recognition device integrating a language error correction model, comprising a module for executing the Lao image text recognition method integrating the language error correction model.

[0061] The beneficial effects of the present invention are:

[0062] 1. This method first uses the Transformer-based basic visual model to obtain Lao text image features, then uses the feature shaping module based on the perceptron model to condense and reduce the extracted text image features to obtain condensed image features, and then uses the encoder layer of the language error correction module based on grammar correction to encode the preliminary recognition results of the visual model to obtain error correction text features, and finally uses the multi-dimensional feature fusion module based on cross attention to fuse the condensed image features and error correction text features to obtain the final recognition results;

[0063] 2. The present invention provides a Lao language error correction model for Lao text image recognition. On the basis of introducing grammatical knowledge, the errors that may occur in daily text image recognition are further corrected. In order to ensure the full use of visual feature information, the feature fusion module performs secondary fusion of visual and language features, thereby enhancing the recognition ability of similar Lao characters.

[0064] 3. The present invention uses a Transformer-based basic visual model to replace the traditional CNN structure, which can model the global information of the entire image by performing global attention calculation on visual features;

[0065] 4. Based on the original character sequence decoding, the present invention introduces a language error correction model, and performs secondary correction on the initial output of the visual model through grammatical knowledge to improve the semantic accuracy of the recognition result. At the same time, before the text features are input into the encoding layer of the language error correction model, a multidimensional feature fusion module based on cross-attention is designed to deeply fuse text and visual features to achieve information complementarity and synergistic enhancement between features. This fusion module effectively captures the potential correlation between text and visual features through the cross-attention mechanism, while improving the feature expression ability, it provides richer contextual information for the error correction model, and ultimately significantly improves the accuracy and robustness of Lao image text recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 It is a structural diagram of a feature shaping module based on a perceptron model in the present invention;

[0067] Figure 2 A model structure diagram of the Laotian image text recognition method integrating a language error correction model in the present invention;

[0068] Figure 3 The figure is an overall flow chart of the Lao image text recognition method integrating the language error correction model in the present invention. DETAILED DESCRIPTION

[0069] Example 1: Figure 1-Figure 3 As shown, a Laotian image text recognition method integrating a language error correction model comprises:

[0070] Step 1. Construction of Lao text image dataset: In view of the lack of public Lao text image dataset in the current field, the present invention constructs a Lao text image dataset through manual collection, annotation and verification to provide data support for the Lao image text recognition method integrating language error correction model.

[0071] Furthermore, the specific steps of Step 1 are:

[0072] Step 1.1. We collected Lao text data from various fields on the Internet, performed preprocessing operations such as deduplication, segmentation, and special character filtering, and built a set of Lao text label datasets for generating Lao text images. In addition, in order to enhance the diversity and authenticity of the dataset, we also obtained Lao text images in real scenes through the Internet.

[0073] Step 1.2, generate corresponding text images from the preprocessed Lao text labels through the text image synthesis algorithm; in order to improve the robustness and generalization ability of the model, the constructed image dataset covers Lao text images with various backgrounds, font colors and different styles, and is divided into training set, validation set and test set;

[0074] Step 1.3: In order to solve the problem of insufficient number of Lao text images in real scenes, data enhancement method is used to improve the diversity of data; this step adopts the following methods:

[0075] Scaling: Adjust the size of Lao text images to simulate different resolutions and scenarios;

[0076] Geometric transformation: geometric transformation of Lao text images by rotation, tilt and translation;

[0077] Occlusion and cropping: Randomly crop or center crop part of the Lao text image;

[0078] Noise: Add random noise to the Lao text image;

[0079] Lighting and contrast adjustment: Simulate the text image characteristics under different lighting environments by adjusting brightness, contrast and saturation.

[0080] Furthermore, in the Step 2, a text representation of Lao text image features and initial text image features is designed based on the Transformer basic visual model, including:

[0081] The basic visual feature extraction module based on Transformer is constructed. The feature extraction of the input Lao text image is performed through the visual Transformer, and the image is converted into a token sequence. This processing method enables the model to assign different attention levels to the entire image or each independent token, thereby mining deeper visual semantic information. In the specific operation, the original image X1∈R is first transformed into H1×W1×C1 Divide into P×P blocks, and then flatten these image blocks into a two-dimensional vector X p ∈R N1×(P2C1) , where P 2 Indicates the resolution of each small block, (P 2C1) represents the number of channels, N1=H1×W1 / P 2 Represents the length of the image block sequence, H1, W1, and C1 are the height, width, and number of channels of the image, respectively; then, a linear mapping layer is used to map the image block to D dimensions as the input of the backbone network encoding layer; similar to the traditional visual Transformer, a D-dimensional learnable classification token is added to extract the global information of the image; at the same time, each image block contains a learnable one-dimensional position embedding, and the final encoder input is expressed as:

[0082]

[0083] Among them, x cls ∈R 1×D Represents classification token, E∈R (N1+1)×D represents the linear mapping matrix, The vector representing the N1th image block, E pos ∈R (N1+1)×D Represents position encoding; Z0 is input into the feature encoder for feature extraction. The feature encoder includes: the multi-head self-attention layer MSA is used to determine the relationship between feature vectors, and the multi-layer perceptron MLP is used to perform feature extraction. The calculation formulas of the two are as follows:

[0084] Z′ m =MSA(LN(Z m-1 ))+Z m-1 ,m=1,…,12

[0085] Z m =MLP(LN(Z′ m ))+Z′ m ,m=1,…,12

[0086] Among them, the multi-layer perceptron consists of two layers of linear activation layers, and the final feature-encoded text image feature Z m ∈R (N1+1)×D Used for subsequent text recognition;

[0087] The text image feature Z m A series of linear projections are performed through the fully connected layer to obtain the text representation of the initial text image features.

[0088] Step 2: Construction of Laotian image text recognition model integrating language error correction model: Construct a language error correction module based on Transformer basic visual model, a feature shaping module based on perceptron model, and a multi-dimensional feature fusion module based on cross attention;

[0089] Design a text representation based on the Transformer-based visual model to extract Lao text image features and initial text image features;

[0090] Design a feature shaping module based on the perceptron model to condense and reduce the dimension of the extracted text image features to obtain condensed image features;

[0091] Design a language error correction module based on grammar correction, whose encoding layer encodes the text representation of the initial text image features recognized by the Transformer basic visual model to obtain the hidden layer representation of the source sequence;

[0092] A multi-dimensional feature fusion module based on cross-attention is designed to use syntactic rule constraint strategy to decode the latent representation of the source sequence and the condensed image features, which is used to deeply mine the text information in Lao text images; this process significantly enhances the recognition accuracy of the model for similar characters in Lao text images;

[0093] Furthermore, in the Step 2, a language error correction module based on grammar correction is designed, and its encoding layer encodes the text representation of the initial text image features recognized based on the Transformer basic visual model to obtain the hidden layer representation of the source sequence, including:

[0094] Construct a language error correction module based on grammatical correction. The basic process of the basic grammatical error correction model includes: first, the text representation X2 of the initial text image features containing grammatical errors is mapped through the encoder to obtain the output of the decoder. The specific formula of the decoder output is:

[0095] H = Encoder(X2, Θ e );

[0096] H is the hidden layer representation of the source sequence, Θ e are the learnable parameters of the encoder;

[0097] Then, the contextual relationship between words and sentence semantics are captured through the multi-head attention mechanism. The calculation formula is:

[0098]

[0099] Among them, Q is the query matrix, K is the parameter matrix, V is the specific value of the queried vector, and d k It is a fixed value used to prevent the output result from being too large;

[0100] The decoder dynamically generates the corrected text through the hidden layer representation and the generated target sequence, and its formula is:

[0101] y t =Decoder(H,Y,Θd ));

[0102] where Y represents the generated partial sequence, and y t represents the next target prediction word, and Θ d are the learnable parameters of the encoder;

[0103] So far, in the inference stage, the decoder generates a preliminary corrected target text according to the hidden layer representation H of the input source sequence through the greedy algorithm:

[0104]

[0105] where N represents the total number of time steps.

[0106] Furthermore, in the Step2, a feature shaping module based on the perceptron model is designed to refine and reduce the dimension of the extracted text image features to obtain refined image features, including:

[0107] Construct a feature shaping module based on the perceptron model. The perceptron is an architecture based on the attention mechanism that can reduce the dimension of high-dimensional inputs such as images and videos without the need for separate adaptation for a specific domain. This module takes cross-attention as the core mechanism, where the initialized query matrix is Q ∈ W (N2×D1) , N2 represents the number of tokens in the current sequence, and D1 represents the channel dimension; at the same time, the high-dimensional matrix to be dimension-reduced is introduced as the key K ∈ W (M×D) and the value V ∈ W (M×C) in the attention formula. M is the number of tokens in the high-dimensional matrix, and C is the channel dimension. At this time, the product of the query Q and the key K is calculated according to the attention formula to obtain an N2 * M matrix, which is further multiplied by the value V to obtain an N2 * C low-dimensional matrix. At this time, since the number of tokens N2 in the query matrix << the number of tokens M in the high-dimensional matrix, a low-dimensional representation that can capture the main feature information of the high-dimensional matrix is finally obtained. To ensure the integrity of high-dimensional feature extraction, the low-dimensional matrix is iteratively updated through multiple self-attention and cross-attention calculations, continuously extracting the information of the high-dimensional matrix. The overall computational complexity is O(N2 * M), but since N2 << M, the computational complexity can be effectively reduced to O(M), which is completely acceptable in practice. At the same time, to avoid losing the semantic relationship between characters before and after during the dimension reduction process, learnable position embeddings are used to label corresponding position tags for each character. After multiple rounds of iterative calculations, this module effectively refines the high-dimensional visual features and finally reduces them to a unified representation suitable for low-dimensional text feature interaction, thus laying a foundation for subsequent multimodal tasks.

[0108] Furthermore, in the Step 2, a multi-dimensional feature fusion module based on cross attention is designed to perform decoding constraints on the latent representation of the source sequence and the condensed image features using a syntactic rule constraint strategy, so as to realize the deep mining of text information in Lao text images, including:

[0109] A cross-attention based multi-dimensional feature fusion module is constructed to fully utilize the visual features extracted by the visual encoder while dynamically adjusting the behavior of the language error correction model. New cross-attention layers are inserted between the encoding layer and the decoding layer of the language correction module based on grammar correction. The keys and values ​​of these attention layers are derived from the condensed image features generated by the feature shaping module, while the query comes from the hidden layer representation H of the output source sequence of the language correction module based on grammar correction. In order to avoid the interference of the fusion module on the original language correction model, in the early stage of training the Lao image text recognition model fused with the language correction model, a tanh gating mechanism is introduced before the output of the multi-dimensional feature fusion module based on cross-attention enters the decoding layer. This mechanism performs a nonlinear transformation on the input through the hyperbolic tangent function tanh, compresses the data range to (-1,1), and is used to capture complex nonlinear relationships, while enhancing the sensitivity to feature changes and suppressing the influence of extreme values. Specifically, the output of the multi-dimensional feature fusion module based on cross-attention is multiplied by the learnable scalar tanh(α) of the gating mechanism and then added to the input representation of the residual connection, where the scalar α is initialized to 0 to ensure that the initial model output is consistent with the pre-trained language correction model, thereby improving the stability of training and the final performance.

[0110] The latent representation of the source sequence and the condensed image features are fused through a multi-dimensional feature fusion module based on cross-attention, and decoded through the decoding end of the language error correction module based on grammar correction to obtain the final output result.

[0111] Step 3: Train the Lao image text recognition model that has been built to integrate the language error correction model;

[0112] Furthermore, the Step 3 includes:

[0113] When training the model, the cross entropy loss function is used as the target optimization function of the Lao image text recognition model that has been built with the fusion language error correction model. The calculation method is shown in the following formula:

[0114] L i =-∑lnP(Y t |M1,θ)(i=v,l)

[0115] Loss = λ v L v +λ l L l

[0116] Among them, M1 is the input Lao text image, θ is the model parameter of the current recognition network, and Y t |M1 is the label corresponding to the tth feature sequence of the Lao text image, L v Represents the loss of the Transformer-based visual model, L l represents the total loss of the language error correction module based on grammar correction, the feature shaping module based on the perceptron model, and the multi-dimensional feature fusion module based on cross attention, λ v , l represents the hyperparameter used to balance the loss and is simply set to 1 in our experiments.

[0117] Step 4, Lao text image recognition: Encapsulate the above training converged model and deploy it on the server. Develop a set of API interfaces to call the Lao image text recognition model that integrates the language error correction model deployed on the server through the API to realize the recognition of Lao text images. In order to improve the efficiency of Lao text image recognition, the relevant parameters of the model input are optimized, batch processing of image data is realized, and the training converged model is encapsulated for deployment on the server. On this basis, a set of API interfaces are developed to realize the fast and efficient recognition of Lao text images.

[0118] Furthermore, the specific steps of Step 4 are:

[0119] Step 4.1: Implement batch processing of the Lao image text recognition model. First, set the height of the model input image to 32 and the maximum width to W. max , for width less than W max The image is filled by pixel padding;

[0120] Step 4.2, save the trained Lao text recognition model as a ".pth" format file, and load the model into memory through a web application based on the Sanic framework;

[0121] Step 4.3. Use the Sanic framework for backend development, encapsulate the decoding process of the Lao text image text recognition model into a callable API interface, and complete the deployment to support multiple concurrent requests of the Web port;

[0122] Step 4.4: Call the Lao text recognition model deployed on the server on the Web side, obtain the recognition result by inputting the Lao text image to be recognized, and display the recognition result on the front-end interface to complete the full process test.

[0123] The present invention provides a Lao image text recognition device integrating a language error correction model, including a module for executing the Lao image text recognition method integrating a language error correction model. Specifically, it may include:

[0124] Lao text image dataset construction module: used to construct a Lao text image dataset through manual collection, annotation and verification;

[0125] The Lao image text recognition model construction module integrated with the language error correction model is used to build a Transformer-based basic visual model, a grammar-correction-based language error correction module, a perceptron-based feature shaping module, and a cross-attention-based multi-dimensional feature fusion module;

[0126] Design a text representation based on the Transformer-based visual model to extract Lao text image features and initial text image features;

[0127] Design a feature shaping module based on the perceptron model to condense and reduce the dimension of the extracted text image features to obtain condensed image features;

[0128] Design a language error correction module based on grammar correction, whose encoding layer encodes the text representation of the initial text image features recognized by the Transformer basic visual model to obtain the hidden layer representation of the source sequence;

[0129] A multi-dimensional feature fusion module based on cross attention is designed to use syntactic rule constraint strategy to decode the latent representation and condensed image features of the source sequence, so as to realize the deep mining of text information in Lao text images.

[0130] Training module: used to train the Lao image text recognition model that has been built with the language error correction model integrated;

[0131] Lao text image recognition module: It is used to encapsulate the above-mentioned trained converged model, deploy it on the server, develop a set of API interfaces, and call the Lao image text recognition model integrated with the language error correction model deployed on the server through the API to realize the recognition of Lao text images.

[0132] In order to verify the effect of the Lao image text recognition model based on multi-granularity decoding constraints proposed in this paper, experiments were conducted on 5 million self-constructed Lao text images. About 30,000 of them were manually annotated, and the rest of the data was synthesized by the algorithm. To ensure fairness, we set unified experimental conditions for all comparative experiments. The input image size was 32×128, the optimizer was Adam, the initial learning rate was set to 1, and the CosineAnnealing strategy was used to gradually reduce the learning rate to ensure that the model has a smaller learning rate when it is most solvable. The batch size was set to 128 and trained for 10 epochs.

[0133] The following comparative experiment was designed for analysis.

[0134] The experiment uses Lao sequence accuracy (SA) as the evaluation indicator, as shown below:

[0135]

[0136] Among them, SA, SL, and LN represent the sequence accuracy, the total number of correct sequences, and the total number of sequences of Lao text image recognition, respectively.

[0137] Experiment 1: Mainly compare the experimental results;

[0138] Table 1 Main experimental results

[0139]

[0140]

[0141] As shown in Table 1, the recognition model proposed in the present invention has achieved an accuracy of 95.3% in the Lao text recognition task, achieving the current optimal effect. Compared with the connectionist temporal classification (CTC) method, the accuracy rate has increased by 8.2%; compared with the sequence-to-sequence (Seq2Seq) method, it has increased by 10.4%. This result shows that this method effectively enhances the ability to extract Lao image features by introducing the visual Transformer encoding module for global context modeling. At the same time, compared with the Transformer-based recognition method, the accuracy rate has increased by 3.6%, proving that the language error correction model can effectively distinguish text characters with similar shapes in Lao during the recognition process and optimize the recognition performance of the model. In addition, compared with the cyclic iterative recognition method based on the language model, the accuracy rate has increased by 1.8%. This further shows that this method not only maintains the recognition accuracy at the character level through the multi-dimensional feature fusion mechanism, but also fully explores the potential information in the visual features and corrects the language model encoder, thereby significantly improving the accuracy and robustness of the overall recognition.

[0142] Experiment 2: ablation experiment results;

[0143] In order to verify the effectiveness of the feature shaping module and the change of feature fusion position, the present invention conducted an ablation experiment on the baseline model. The experimental results are shown in Table 2 ("×" represents not adopted, "√" represents adopted):

[0144] Table 2 shows the influence of feature shaping module and feature fusion position on recognition

[0145]

[0146] As shown in Table 2, from the experimental results, it can be seen that the introduction of the feature shaping module can effectively improve the performance of the fusion module, thereby improving the accuracy of model recognition; at the same time, changing the feature fusion position can also effectively improve the utilization of visual features and improve the main performance of the model. This proves the effectiveness of the proposed method.

[0147] Experiment 3: Experimental results on the real test set;

[0148] In order to verify the performance of the invented model in terms of training and inference speed, experiments were conducted on a dataset of 1,000 real-scene Lao text images.

[0149] Table 3 shows the experimental results on the real test set.

[0150]

[0151] As shown in Table 3, the proposed method achieved a recognition accuracy of 93.2% in real scenarios, significantly better than other comparison methods. Among them, the accuracy of the ResNet50+BiLSTM+CTC method was 88.6%, the ResNet50+Transformer method was 90.8%, and the ABINet method was 92.4%. The advantages of the proposed method in multi-dimensional feature fusion and global context modeling enable it to achieve the highest recognition accuracy in Lao text recognition tasks, further verifying its robustness and effectiveness in complex scenarios.

[0152] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A Laotian image text recognition method integrating a language error correction model, characterized by: The method comprises: Step 1: Construction of Lao text image dataset: A Lao text image dataset is constructed by manual collection, annotation and verification; Step 2: Construction of Lao image text recognition model integrating language error correction model: Construct a language error correction module based on Transformer basic visual model, a feature shaping module based on perceptron model, and a multi-dimensional feature fusion module based on cross attention; Design a text representation based on the Transformer-based visual model to extract Lao text image features and initial text image features; Design a feature shaping module based on the perceptron model to condense and reduce the dimension of the extracted text image features to obtain condensed image features; Design a language error correction module based on grammar correction, whose encoding layer encodes the text representation of the initial text image features recognized by the Transformer basic visual model to obtain the hidden layer representation of the source sequence; A multi-dimensional feature fusion module based on cross attention is designed to use syntactic rule constraint strategy to decode the latent representation and condensed image features of the source sequence, so as to realize the deep mining of text information in Lao text images. Step 3: Train the Lao image text recognition model that has been built to integrate the language error correction model; Step 4, Lao text image recognition: Encapsulate the above trained converged model and deploy it on the server, develop a set of API interfaces, and call the Lao image text recognition model integrated with the language error correction model deployed on the server through the API to realize the recognition of Lao text images.

2. The Lao image text recognition method integrating the language error correction model according to claim 1 is characterized in that: The specific steps of Step 1 are: Step 1.1, collect Lao text data from the Internet, perform deduplication, segmentation and special character filtering preprocessing operations, and build a set of Lao text label data sets for generating Lao text images; also obtain Lao text images in real scenes through the network; Step 1.2, generate corresponding text images from the preprocessed Lao text labels through a text-image synthesis algorithm; the constructed image dataset covers Lao text images of various backgrounds, font colors and different styles, and is divided into training set, validation set and test set; Step 1.3: Data enhancement methods are used to improve data diversity. This step uses the following methods: Scaling: Adjust the size of Lao text images to simulate different resolutions and scenarios; Geometric transformation: geometric transformation of Lao text images by rotation, tilt and translation; Occlusion and cropping: Randomly crop or center crop part of the Lao text image; Noise: Add random noise to the Lao text image; Lighting and contrast adjustment: Simulate the text image characteristics under different lighting environments by adjusting brightness, contrast and saturation.

3. The Lao image text recognition method integrating the language error correction model according to claim 1 is characterized in that: In the Step 2, a text representation of Lao text image features and initial text image features is designed based on the Transformer basic visual model, including: Construct a basic visual feature extraction module based on Transformer, extract features from the input Lao text image through the visual Transformer, and convert the image into a token sequence; in the specific operation, first use the linear mapping layer to transform the original image X1∈R H1×W1×C1 Divide into P×P blocks, and then flatten these image blocks into two-dimensional vectors Where P 2 Indicates the resolution of each small block, (P 2 C1) represents the number of channels, N1=H1×W1 / P 2 Represents the length of the image block sequence, H1, W1, and C1 are the height, width, and number of channels of the image, respectively; then, a linear mapping layer is used to map the image block to D dimensions as the input of the backbone network encoding layer; similar to the traditional visual Transformer, a D-dimensional learnable classification token is added to extract the global information of the image; at the same time, each image block contains a learnable one-dimensional position embedding, and the final encoder input is expressed as: Among them, x cls ∈R 1×D Represents classification token, E∈R (N1+1)×D represents the linear mapping matrix, The vector representing the N1th image block, E pos ∈R (N1+1)×D Represents position encoding; Z0 is input into the feature encoder for feature extraction. The feature encoder includes: the multi-head self-attention layer MSA is used to determine the relationship between feature vectors, and the multi-layer perceptron MLP is used to perform feature extraction. The calculation formulas of the two are as follows: Z′ m =MSA(LN(Z m-1 ))+Z m-1 ,m=1,…,12 WITH m =MLP(LN(Z′ m ))+Z′ m ,m=1,…,12 Among them, the multi-layer perceptron consists of two layers of linear activation layers, and the final feature-encoded text image feature Z m ∈R (N1 +1)×D Used for subsequent text recognition; The text image feature Z m A series of linear projections are performed through the fully connected layer to obtain the text representation of the initial text image features.

4. The Lao image text recognition method integrating the language error correction model according to claim 1 is characterized in that: In the Step 2, a language error correction module based on grammar correction is designed, and its encoding layer encodes the text representation of the initial text image features recognized based on the Transformer basic visual model to obtain the hidden layer representation of the source sequence, including: Construct a language error correction module based on grammatical correction. The basic process of the basic grammatical error correction model includes: first, the text representation X2 of the initial text image features containing grammatical errors is mapped through the encoder to obtain the output of the decoder. The specific formula of the decoder output is: H=Encoder(X2,Θ e ); H is the hidden layer representation of the source sequence, Θ e are the learnable parameters of the encoder; Then, the contextual relationship between words and sentence semantics are captured through the multi-head attention mechanism. The calculation formula is: Among them, Q is the query matrix, K is the parameter matrix, V is the specific value of the queried vector, and d k It is a fixed value used to prevent the output result from being too large; The decoder dynamically generates the corrected text through the hidden layer representation and the generated target sequence, and its formula is: y t =Decoder(H,Y,Θ d ); Among them, Y represents the generated partial sequence, y t represents the next target prediction word, Θ d are the learnable parameters of the encoder; At this point, in the inference phase, the decoder generates a preliminary corrected target text through a greedy algorithm based on the hidden layer representation H of the input source sequence: Where N represents the total number of time steps.

5. The Lao image text recognition method integrating the language error correction model according to claim 1 is characterized in that: In the Step 2, a feature shaping module based on a perceptron model is designed to condense and reduce the dimension of the extracted text image features to obtain condensed image features; including: Construct a feature shaping module based on the perceptron model, which uses cross attention as the core mechanism, where the initialized query matrix is ​​Q∈W (N2×D1) , N2 represents the number of tokens in the current sequence, D1 represents the channel dimension; at the same time, the high-dimensional matrix to be reduced is used as the key K∈W in the attention formula (M×D) The sum V∈W (M×C) In this example, M is the number of tokens in the high-dimensional matrix, and C is the channel dimension. At this time, the product of the query Q and the key K is calculated according to the attention formula to obtain an N2*M matrix, which is further multiplied by the value V to obtain an N2*C low-dimensional matrix.

6. The Laotian image text recognition method integrating the language error correction model according to claim 1 is characterized in that: In the Step 2, a multi-dimensional feature fusion module based on cross attention is designed to perform decoding constraints on the hidden layer representation and condensed image features of the source sequence using a syntactic rule constraint strategy, so as to realize the deep mining of text information in Lao text images, including: A multi-dimensional feature fusion module based on cross-attention is constructed. A new cross-attention layer is inserted between the encoding layer and the decoding layer of the language error correction module based on grammar correction. The keys and values ​​of these attention layers are derived from the condensed image features generated by the feature shaping module, while the query comes from the hidden layer representation H of the output source sequence of the language error correction module based on grammar correction. In the early stage of training the Lao image text recognition model fused with the language error correction model, a tanh gating mechanism is introduced before the output of the multi-dimensional feature fusion module based on cross-attention enters the decoding layer. This mechanism performs a nonlinear transformation on the input through the hyperbolic tangent function tanh, compressing the data range to (-1,1) to capture complex nonlinear relationships, while enhancing the sensitivity to feature changes and suppressing the influence of extreme values. Specifically, the output of the multi-dimensional feature fusion module based on cross-attention is multiplied by the learnable scalar tanh(α) of the gating mechanism and then added to the input representation of the residual connection, where the scalar α is initialized to 0 to ensure that the initial model output is consistent with the pre-trained language error correction model. The latent representation of the source sequence and the condensed image features are fused through a multi-dimensional feature fusion module based on cross-attention, and decoded through the decoding end of the language error correction module based on grammar correction to obtain the final output result.

7. The Lao image text recognition method integrating the language error correction model according to claim 1 is characterized in that: The Step 3 includes: When training the model, the cross entropy loss function is used as the target optimization function of the Lao image text recognition model that has been built with the fusion language error correction model. The calculation method is shown in the following formula: L i =-∑lnP(Y t |M1,θ)(i=v,l) Loss=λ v L v +λ l L l Among them, M1 is the input Lao text image, θ is the model parameter of the current recognition network, and Y t |M1 is the label corresponding to the tth feature sequence of the Lao text image, L v Represents the loss of the Transformer-based visual model, L l represents the total loss of the language error correction module based on grammar correction, the feature shaping module based on the perceptron model, and the multi-dimensional feature fusion module based on cross attention, λ v , l represents the hyperparameter used to balance the loss.

8. The Lao image text recognition method integrating the language error correction model according to claim 1 is characterized in that: The specific steps of Step 4 are: Step 4.1: Implement batch processing of the Lao image text recognition model. First, set the height of the model input image to 32 and the maximum width to W. max , for width less than W max The image is filled by pixel padding; Step 4.2, save the trained Lao text recognition model as a ".pth" format file, and load the model into memory through a web application based on the Sanic framework; Step 4.

3. Use the Sanic framework for backend development, encapsulate the decoding process of the Lao text image text recognition model into a callable API interface, and complete the deployment to support multiple concurrent requests of the Web port; Step 4.4: Call the Lao text recognition model deployed on the server on the Web side, obtain the recognition result by inputting the Lao text image to be recognized, and display the recognition result on the front-end interface to complete the whole process test.

9. A Laotian image text recognition device integrating a language error correction model, characterized in that: The method comprises a module for executing the Lao image text recognition method integrating the language error correction model as claimed in any one of claims 1 to 8.

Images