Lao text image recognition method and device based on graph network topology correction

Abstract

The invention relates to a Lao text image text recognition method and device based on graph network topology correction, and belongs to the field of natural language processing. The invention provides a Lao text image recognition method based on graph network topology correction in order to solve the problem that superscript and superscript of characters in Lao text images are lost in view of Lao text images widely existing in Lao texts. Lao text image data set construction and preprocessing, design of a graph network topology feature extraction module based on language constraints, a cross-modal feature fusion module based on a gating mechanism and Lao text image recognition are mainly performed. Lao text image recognition devices based on graph network topology correction are developed in a modularized mode according to the four parts of functions, and the problem that Lao characters are wrongly recognized and missed to recognize due to complex distribution of superscript and superscript is effectively solved.

Description

Technical Field

[0001] This invention provides a method and apparatus for Lao text image recognition based on graph network topology correction, belonging to the field of natural language processing technology. Background Technology

[0002] Lao uses a syllable-based writing system, where characters consist not only of consonants and vowels but also tone marks, which can appear above, below, left, or right of consonants. This non-linear, two-dimensional character arrangement makes Lao visually far more complex than the linear alphabetical arrangement of English. The unpredictable spatial distribution of characters further complicates character location and recognition.

[0003] Existing visual Transformer-based text recognition models have significantly alleviated the confusion problem of similar characters in Lao through global self-attention mechanisms and collaborative optimization with language models. However, they still face fundamental challenges in handling fine-grained subscript and superscript structures. Subscripts and superscripts are widely present in Lao texts, such as... Figure 4 As shown, they are typically distributed at discrete positions along the edges of the main character and are very small in size (often less than 1 / 5 of the main character area, such as consonants). With vowel symbols Combining to form syllables The geometric sparsity and semantic dependence of such symbols pose two key challenges to the existing Transformer architecture: First, the standard image segmentation strategy will separate tiny subscripts and superscripts from their main characters into different image blocks, resulting in the weakening of local contextual associations during the embedding stage; Second, the global computation paradigm of the self-attention mechanism tends to focus on high-frequency main features, while lacking sensitivity to low-frequency details in edge regions, which further exacerbates the risk of feature annihilation of subscripts and superscripts.

[0004] To address the above problems, this invention proposes a Lao text image recognition method and apparatus based on graph network topology correction. Summary of the Invention

[0005] This invention proposes a Lao text image recognition method and apparatus based on graph network topology correction, which is used to solve the problem of misidentification and omission caused by the complex distribution of superscripts and subscripts in Lao text images, and improve the accuracy and robustness of Lao text image recognition technology.

[0006] The technical solution of this invention is: a Lao text image recognition method based on graph network topology correction, the method comprising:

[0007] Step 1: Construction and preprocessing of Lao text image dataset: Lao text images with various backgrounds, fonts and styles are constructed by combining manual collection, algorithm synthesis and data augmentation.

[0008] Step2、design a language constraint-based graph network topology feature extraction module: adopt a character topology graph construction method based on linguistic constraints and a global graph reasoning method based on graph convolution network to construct and condense the feature map output by the Transformer-based image encoder, and realize the extraction of the graph network topology feature;

[0009] Step3、design a cross-modal feature fusion module based on a gating mechanism: for heterogeneous features such as visual features, text features, and graph network topology features, design a two-stage gating fusion mechanism; in the first stage, fuse visual features and text features through cross-modal attention and gating mechanism; in the second stage, use dimension projection alignment method and double-modal gating mechanism to dynamically balance the weight of graph network topology features and visual-semantic fusion features, and realize the deep fusion of cross-modal features;

[0010] Step4、Laotian text image recognition: encapsulate and deploy the Laotian text image recognition model based on graph network topology correction including the modules in Step2 and Step3 to the server side, design a unified API interface, and realize automatic recognition and result return of Laotian text images.

[0011] Further, the specific steps of Step1 are:

[0012] Step1.1, obtain original Laotian text images through network crawling, cover multiple environments, and ensure data diversity; at the same time, generate synthetic images based on the conditional diffusion model, and construct combinations including base characters and attached superscript and subscript symbols;

[0013] Step1.2, label the collected images with text regions and the corresponding Laotian text content to ensure character-level alignment; use automatic filtering and manual verification mechanisms to remove low-quality, blurred, or incorrectly labeled text images;

[0014] Step1.3, use multiple data augmentation strategies to improve data diversity and model robustness; including:

[0015] Scaling: adjust the image size to simulate different resolutions;

[0016] Geometric transformation: rotate, tilt, and translate;

[0017] Occlusion and cropping: randomly or centrally crop part of the image area;

[0018] Noise injection: add random noise interference;

[0019] Lighting and Contrast Adjustment: Adjust brightness, contrast, and saturation to enhance the image's adaptability under different lighting conditions;

[0020] Step 1.4: Finally, construct a large-scale dataset, covering various text formats, and divide it into training set, validation set and test set.

[0021] Furthermore, in Step 2, the language-constrained graph network topology feature extraction module includes two parts: feature construction and feature condensation.

[0022] The feature construction part adopts a character topology graph construction method based on linguistic constraints. The feature map output by the Transformer-based image encoder is used to construct character clusters through spatial similarity clustering. Then, the linguistic constraints are transformed into a graph topology structure. Local subgraphs are generated with the geometric center as the root node and linked into a global graph to realize the construction of graph network topology features.

[0023] The feature condensation part adopts a global graph reasoning method based on graph convolutional networks, which performs hierarchical modeling and information aggregation of local and global structures, and performs feature condensation through stacked graph convolution and graph pooling operations, thereby realizing the extraction of graph network topological features.

[0024] Furthermore, in Step 2, the character topology graph construction method based on linguistic constraints constructs graph network topology features; including:

[0025] By introducing the concept of Lao character clusters, the morphological co-occurrence relationship between base characters and subscripts and superscripts is explicitly modeled, and linguistic constraints are transformed into graph topology. Here, a character cluster is defined as the smallest semantic unit composed of a base character and its attached subscripts and superscripts.

[0026] Then, the inductive bias of the graph network is used to capture the spatial-semantic dependencies inside and outside the character cluster, which alleviates feature ambiguity caused by subscript or superscript offset or fuzziness.

[0027] Furthermore, in the specific operation, the first stage is character cluster segmentation and root node localization. A geometry-driven clustering algorithm is used to divide character cluster regions based on the spatial density and directional consistency of character pixels. Within each cluster, the geometric center (xi) of the character cluster is used as the reference point. c ,y c ) is the root node v root The remaining pixel nodes are assigned based on their relative coordinates. Let the coordinates of the i-th pixel node be (x, y, y). i ,y i ), root node v root The coordinates are (x c ,y c If ), then the Euclidean distance d from the pixel node to the root node is... i The calculation process is as follows:

[0028] d i = ||(x i -y j )-(x c ,y c )||2

[0029] In the local subgraph construction stage, a weighted subgraph is constructed within the character cluster where the edge weight w ij is determined by both the node feature similarity and the spatial proximity, and the calculation process is as follows:

[0030] w ij = σ(cos(h i ,h j )·exp(-γd ij ))

[0031] where h i , h j are the node visual features, cos() is the cosine similarity, γ is the distance attenuation coefficient, σ is the activation function, and d ij is the Euclidean distance between node i and node j, used to measure the proximity of the two nodes in the spatial position.

[0032] Only strong correlation edges with w ij > 0.7 are retained to suppress noise interference; after the construction of the local subgraph , the root nodes of the local subgraphs are connected in sequence to link them to the 2-level complete graph, completing the connection of the global subgraph G global = (V, E).

[0033] Further, in Step 2, the global graph inference method based on graph convolution network is used to condense the graph network topology features; including:

[0034] First, on the initial global subgraph G global = (V, E), the graph convolution network is used to aggregate node features layer by layer, and the calculation process is as follows:

[0035]

[0036] where E ij is the edge weight in the adjacency matrix, W (l) is the learnable weight matrix, σ is the activation function, is the feature vector of node i after the l-th layer of graph convolution, and when l = 0, it is the visual feature, and when l ≥ 0, it is the abstract feature fused with the topological relationship; is the updated feature vector of node i after the (l+1)-th layer of graph convolution.

[0037] After each convolution, the differentiable pooling is used to dynamically compress the redundant nodes, and the key topological structure is reserved by the node importance score, and the calculation process is as follows:

[0038]

[0039] The pooling matrix The number of nodes is reduced from N l to N l+1 , and the low-contribution nodes are gradually removed; after L-layer iteration, the feature of each character cluster subgraph is compressed to the root node v root , forming a concise graph network topological feature H global ∈R K×d , wherein K is the number of decoded characters, that is, the total number of character clusters, and d is the dimension of each character feature, and the output dimension from the Transformer encoder;

[0040] This process realizes multi-granularity feature expression through hierarchical abstraction: the local level models the morphological symbiotic relationship between base characters and superscripts and subscripts, and the global level captures the reading order and semantic coherence between character clusters.

[0041] Further, the Step 3 comprises:

[0042] First, the visual Transformer-based text recognition model extracts the visual features H vis ∈R N×d of the input image, wherein N represents the length of the visual feature sequence, and d represents the dimension of the feature; then the visual feature is sequentially decoded to obtain a preliminary recognition result The is input into the encoder of the grammar error correction model GEC, and the text feature H text ∈R M×d is output, wherein M represents the length of the text feature sequence;

[0043] In order to fuse the visual and text features, a cross-modal attention mechanism is used:

[0044]

[0045] Wherein, Q=H text W Q , K=H vis W K , V=H vis W V , W Q ∈R d×dk , W K ∈R d×dk , W V ∈R d×dk are learnable attention weight matrices, and dk is the key dimension of the attention mechanism.

[0046] After that, the first stage gating mechanism is used to generate visual-semantic fusion features:

[0047] G1 = σ(W g1 [H vis ; H text ]+b g1 )

[0048] H vs = G1⊙H vis +(1-G1)⊙Attention(Q, K, V)

[0049] where W g1 ∈R 2d×d is the gating weight matrix of the first stage, b g1 ∈R d is the gating bias vector of the first stage, G1∈R N×d is the gating coefficient of the first stage, which is used to dynamically balance the weights of the original visual features and cross-modal attention features; H vs ∈R N×d is the visual-semantic fusion feature, σ is the activation function, and ⊙ represents element-wise multiplication.

[0050] The generated visual-semantic fusion feature H vs ∈R L×d is adjusted to be consistent with the graph node sequence dimension through linear projection:

[0051] H′ vs = W p H vs +b p

[0052] where W p ∈R N×K is the dimension projection weight matrix, b p ∈R K is the projection bias vector, H′ vs ∈R K×d is the aligned visual-semantic fusion feature after linear projection adjustment, and K is the number of character clusters.

[0053] The second stage gating mechanism is designed to dynamically balance the contributions of the graph network topology feature H global ∈R K×d and the aligned visual-semantic feature H′ vs ∈R K×d , and the specific calculation formula is as follows:

[0054] G2 = σ(W g2 [H global ; H′ vs ]+bg2 )

[0055] H fused =G2⊙tanh(W t H global )+(1-G2)⊙H′ vs

[0056] where W g2 ∈R 2d×d is the second-stage gating weight matrix, b g2 ∈R d is the second-stage gating bias vector, G2∈R K×d is the second-stage gating coefficient, used to dynamically adjust the fusion weight of the graph network topology feature and the visual-semantic feature; H fused ∈R K×d is the final fusion output feature, W t ∈R d×d is the graph network feature transformation matrix, and σ is an activation function, and ⊙ represents element-wise multiplication.

[0057] Finally, the fusion output feature H fuse is input into a fully connected layer to generate the final recognition result.

[0058] The cross-modal feature fusion module based on the gating mechanism realizes the deep fusion of cross-modal features through a two-stage gating design: in the first stage, the gating mechanism is used to fuse the visual features and the text features to generate visual-semantic fusion features; in the second stage, the cross-modal dimension adaptation method based on projection alignment is used to solve the problem of heterogeneous feature dimension mismatch, and a dual-modal gating mechanism is designed to dynamically balance the weight of the graph network topology feature and the visual-semantic fusion feature, so as to realize the dynamic allocation of fine-grained feature weight.

[0059] During model training, the output result of the graph network topology feature extraction module based on language constraint is supervised by a character cluster classification loss L cls , the output result of the cross-modal feature fusion module is supervised by a sequence alignment loss L ctc , and the output result of the whole model is supervised by a whole loss L total . The calculation formula is as follows:

[0060]

[0061] L ctc =-log P(y|H fused )

[0062] L total =αL cls +βL ctc

[0063] where K is the number of character clusters, C is the number of character cluster categories, y k,cy is the true label of the kth character cluster, p k,c is the probability that the kth character cluster predicted by the model belongs to class c; y is the true character sequence label, L total represents the total loss of the model, and alpha and beta represent hyperparameters for balancing the loss, P(y|H fused represents the probability of predicting the true character sequence y given the fused feature sequence H fused .

[0064] Further, the specific steps of Step4 are:

[0065] Step4.1, first, the Laos text image to be recognized is uniformly sized, and the specific operation is: the height of the input image is fixed to 32 pixels, and the width is set to W max , for images with a width less than W max , the size is padded by padding pixels on the edge to ensure that all input images meet the uniform input specification, facilitating batch processing and efficient recognition of the model;

[0066] Step4.2, save the model weight of the Laos text image recognition model based on graph network topology correction trained and converged in the early stage as a ".pth" format model file; then, based on the Sanic framework, a web service environment is built, and the model weight file is loaded into the server memory to realize fast calling and online recognition support of the model;

[0067] Step4.3, relying on the Sanic framework to complete the development of the backend service, encapsulating the key processes of the model, designing standard API interfaces, and realizing unified calling of the Laos text image recognition service; at the same time, the service is deployed to the server to support multiple concurrent access requests on the web side;

[0068] Step4.4, in the front-end page, the Laos text image to be recognized is input by calling the API interface deployed on the server side, triggering the recognition process; after the server side completes the image recognition and result prediction, the recognition result is returned to the front-end and visualized on the page, finally realizing complete process verification and function test of the Laos text image recognition system.

[0069] The present application also provides a Laos text image recognition device based on graph network topology correction, comprising a module for executing the Laos text image recognition method based on graph network topology correction.

[0070] The present application has the following advantages:

[0071] 1. The method first constructs a subgraph by dividing the feature map output by the Transformer-based image encoder into subgraph regions according to the Lao character cluster structure based on spatial similarity, and links the root nodes to construct a global graph according to node similarity, to realize graph topology feature construction; then uses a multi-scale graph convolutional network to model and aggregate information at different levels, and performs feature condensation through stacked graph convolution and graph pooling operations, to realize fine-grained extraction of text image features; finally, through a feature fusion module based on feature dimension alignment and dynamic weight matching, the graph topology features and visual-linguistic fusion features are dynamically aligned to realize deep fusion of three-dimensional features in the Lao text image, and the final recognition result is obtained;

[0072] 2. The application provides a Lao text image recognition model constructed for the misrecognition and missed recognition of Lao text images caused by complex distribution of superscripts and subscripts, which explicitly models the geometric and semantic association of character clusters through spatial similarity clustering, combines a graph convolutional network to condense topology features layer by layer, and designs a gating fusion mechanism to dynamically balance the weights of graph topology features and visual-linguistic features, to finally realize efficient fusion of cross-modal features;

[0073] 3. The application uses a Transformer-based visual model instead of a traditional CNN structure, which can model the global information of the whole image by performing global attention calculation on visual features; and combines a grammar correction model to effectively alleviate the misjudgment problem of similar characters by the visual model;

[0074] 4. The application adds a topology correction module based on a graph network to a Transformer-based text recognition model to enhance the robust representation of edge features by explicitly modeling the character-superscript and subscript-topology relationship, and improve the semantic accuracy of the recognition result. For the unique character structure of low-resource languages such as Lao (e.g., fine-grained superscripts and subscripts), an extensible technical framework is proposed to significantly improve the accuracy and robustness of Lao scene image text recognition. BRIEF DESCRIPTION OF DRAWINGS

[0075] Figure 1 The graph convolutional network inference architecture in the application;

[0076] Figure 2 The Lao text scene image recognition model framework based on graph network topology correction in the application;

[0077] Figure 3 The overall flowchart of the Lao text scene image recognition method based on graph network topology correction in the application;

[0078] Figure 4An example of a complex character combination for the Lao language. DETAILED DESCRIPTION

[0079] Embodiment 1: As shown in the Lao language text image recognition method based on graph network topology correction, the method comprises: Figures 1-3

[0080] Step 1, Lao language text image dataset construction and preprocessing: In order to solve the problem of lack of Lao language text image data in low resource scenarios, a combination of manual collection, algorithm synthesis and data enhancement is used to construct a Lao language text image containing 5 million images of various backgrounds, fonts and styles;

[0081] Step 2, design of graph network topology feature extraction module based on language constraints: In order to solve the semantic break problem of traditional segmentation model caused by the topological heterogeneity of Lao language characters, a character topology graph construction method based on linguistic constraints and a global graph reasoning method based on graph convolution network are used to construct and condense the feature map output by the image encoder based on Transformer, and the extraction of graph network topology features is realized;

[0082] Step 3, design of cross-modal feature fusion module based on gating mechanism: For heterogeneous features such as visual features, text features and graph network topology features, a two-stage gating fusion mechanism is designed; In the first stage, cross-modal attention and gating mechanism are used to fuse visual features and text features; In the second stage, dimension projection alignment method and double modal gating mechanism are used to dynamically balance the weight of graph network topology features and visual-semantic fusion features, realizing the deep fusion of cross-modal features;

[0083] Step 4, Lao language text image recognition: The Lao language text image recognition model based on graph network topology correction including the modules in Step 2 and Step 3 is encapsulated and deployed to the server side, a unified API interface is designed, and automatic recognition and result return of Lao language text image are realized.

[0084] Further, the specific steps of Step 1 are as follows:

[0085] Step 1.1, obtain the original Lao language text image through network crawling and other methods, cover multiple environments such as natural scenes, documents and billboards, ensure data diversity; At the same time, generate synthetic images based on conditional diffusion model, construct combinations including base characters and attached superscript and subscript symbols (for example, auxiliary and vowel symbols combine to form syllables ), effectively expand the data size and improve the sample richness;

[0086] ​Step 1.2. Text region annotation (such as quadrilateral frame or pixel-level segmentation annotation) is performed on the collected images, and the corresponding Lao language text content is labeled to ensure character-level alignment. Automatic filtering and manual verification mechanisms are used to remove low-quality, blurred, or incorrectly labeled text images.

[0087] Step 1.3. To improve data diversity and model robustness, various data augmentation strategies are further employed to enhance data diversity and model robustness. These include:

[0088] Scaling: Adjusting image size to simulate different resolutions.

[0089] Geometric transformation: Performing rotation, tilt, and translation operations.

[0090] Occlusion and cropping: Randomly or centrally cropping part of the image area.

[0091] Noise injection: Adding random noise interference.

[0092] Lighting and contrast adjustment: Adjusting brightness, contrast, and saturation to enhance image adaptability under different lighting conditions.

[0093] Step 1.4. Finally, a large-scale dataset containing 5 million images is constructed, covering natural scenes, printed bodies, handwritten bodies, and other text forms, and is divided into training, validation, and test sets.

[0094] Further, in Step 2, the language constraint-based graph network topology feature extraction module includes two parts: feature construction and feature condensation.

[0095] The feature construction part adopts a language constraint-based character topology graph construction method. The feature map output by the Transformer-based image encoder is clustered by spatial similarity to construct character clusters. Then, the linguistic constraints are converted into graph topology structures. The geometric center is taken as the root node to generate local subgraphs and link them into a global graph, realizing the construction of graph network topology features.

[0096] The feature condensation part adopts a global graph reasoning method based on graph convolutional networks. The local and global structures are hierarchically modeled and information is aggregated. Through stacked graph convolution and graph pooling operations, feature condensation is performed, and the extraction of graph network topology features is realized, effectively solving the semantic breakage problem of traditional segmentation models caused by the topological heterogeneity of Lao characters.

[0097] Further, in Step 2, the language constraint-based character topology graph construction method is used to construct graph network topology features. This includes:

[0098] By introducing the concept of Lao character cluster, explicitly modeling the morphological symbiotic relationship between base characters and their superscripts / subscripts, and converting linguistic constraints into graph topology, where a character cluster is defined as the semantic minimum unit composed of a base character and its attached superscript / subscript, for example

[0099] The spatial-semantic dependencies within and outside the character cluster are captured by the inductive bias of the graph network, alleviating the feature ambiguity caused by superscript / subscript shift or ambiguity.

[0100] Further, in specific operations, first is the character cluster segmentation and root node positioning stage, a geometry-driven clustering algorithm is adopted to divide the character cluster area based on the spatial density and direction consistency of character pixels; the geometric center (x c ,y c ) of the character cluster is taken as the root node v root in each cluster, and the remaining pixel nodes are divided according to the relative position of the coordinates, and the coordinates of the i-th pixel node are (x i ,y i ), and the coordinates of the root node v root are (x c ,y c ), then the Euclidean distance d i from the pixel node to the root node is calculated as follows:

[0101] d i =||(x i -y j )-(x c ,y c )||2

[0102] In the local subgraph construction stage, a weighted subgraph G is constructed within the character cluster, where the edge weight w ij in the adjacency matrix E is determined by the similarity of node features and spatial proximity, and the calculation process is as follows:

[0103] w ij =σ(cos(h i ,h j )·exp(-γd ij ))

[0104] where h i and h j are node visual features, cos() is cosine similarity, γ is distance decay coefficient, σ is activation function, and d ij is the Euclidean distance between node i and node j, which measures the proximity of two nodes in spatial position;

[0105] Only strong correlation edges with w ij >0.7 are retained to suppress noise interference; after completing the local subgraph After the construction of the local sub-graphs, the global sub-graph G global is completed by connecting the root nodes of the local sub-graphs to the 2-level complete graph in sequence.

[0106] Further, in Step 2, the global graph inference method based on the graph convolution network is used to condense the graph network topology features, including:

[0107] First, on the initial global sub-graph G global =(V, E), the graph convolution network is used to aggregate node features layer by layer, and the calculation process is as follows:

[0108]

[0109] where E ij is the edge weight in the adjacency matrix, W (l) is a learnable weight matrix, σ is an activation function, is the feature vector of node i after the l-th layer of graph convolution, when l = 0, it is the visual feature, and when l ≥ 0, it is the abstract feature fused with the topological relationship; is the updated feature vector of node i after the (l+1)-th layer of graph convolution;

[0110] After each layer of convolution, a differentiable pooling is used to dynamically compress redundant nodes, and key topological structures are preserved through node importance scoring, and the calculation process is as follows:

[0111]

[0112] The pooling matrix compresses the number of nodes from N l to N l+1 , gradually eliminating low-contribution nodes; after L layers of iteration, the features of each character cluster sub-graph are compressed to the root node v root , forming the condensed graph network topology features H global ∈R K×d , where K is the number of decoded characters, i.e., the total number of character clusters, and d is the dimension of each character feature, which comes from the output dimension of the Transformer encoder;

[0113] This process realizes multi-granularity feature expression through hierarchical abstraction: local level modeling of the co-occurrence relationship between base characters and superscripts and subscripts, and global level capturing of reading order and semantic coherence between character clusters.

[0114] Further, Step 3 includes:

[0115] First, the visual Transformer-based text recognition model extracts the visual features H vis ∈RN×d where N denotes the length of the visual feature sequence, d denotes the dimension of the feature; and the visual feature is sequentially decoded to obtain a preliminary recognition result input into an encoder of a syntax error correction model GEC, and output text features H text ∈R M×d where M denotes the length of the text feature sequence;

[0116] In order to fuse the visual and text features, a cross-modal attention mechanism is adopted:

[0117]

[0118] where Q = H text W Q , K = H vis W K , V = H vis W V , W Q ∈R d×dk , W K ∈R d×dk , W V ∈R d×dk is a learnable attention weight matrix, and dk is the key dimension of the attention mechanism;

[0119] Subsequently, a first-stage gating mechanism is adopted to generate visual-semantic fusion features:

[0120] G1 = σ(W g1 [H vis ; H text ] + b g1 )

[0121] H vs = G1 ⊙ H vis + (1-G1) ⊙ Attention(Q, K, V)

[0122] where W g1 ∈R 2d×d is a gating weight matrix of the first stage, b g1 ∈R d is a first-stage gating bias vector, G1 ∈R N×d is a first-stage gating coefficient, used to dynamically balance the weights of the original visual features and the cross-modal attention features; H vs ∈R N×d is a visual-semantic fusion feature, and σ is an activation function, and ⊙ represents element-wise multiplication;

[0123] The generated visual-semantic fusion features H vs ∈R L×d ​Adjust to the graph node sequence dimension by linear projection:

[0124] H′ vs = W p H vs + b p

[0125] wherein, W p ∈R N×K is a dimension projection weight matrix, b p ∈R K is a projection bias vector, H′ vs ∈R K×d is the aligned visual-semantic fusion feature after linear projection adjustment, and K is the number of character clusters;

[0126] A second stage gating mechanism is designed to dynamically balance the contribution of the graph network topology feature H global ∈R K×d and the aligned visual-semantic feature H′ vs ∈R K×d , and the specific calculation formula is as follows:

[0127] G2 = σ(W g2 [H globtl ; H′ vs ] + b g2 )

[0128] H fused = G2 ⊙ tanh(W t H global ) + (1-G2) ⊙ H′ vs

[0129] wherein, W g2 ∈R 2d×d is a second stage gating weight matrix, b g2 ∈R d is a second stage gating bias vector, G2 ∈R K×d is a second stage gating coefficient, used to dynamically adjust the fusion weight of the graph network topology feature and the visual-semantic feature; H fused ∈R K×d is the final fusion output feature, W t ∈R d×d is a graph network feature transformation matrix, σ is an activation function, and ⊙ represents element-wise multiplication.

[0130] Finally, the fusion output feature H fuse is input into a fully connected layer to generate the final recognition result.

[0131] The cross-modal feature fusion module based on the gating mechanism realizes the deep fusion of cross-modal features through a two-stage gating design: in the first stage, the gating mechanism is used to fuse visual features and text features to generate visual-semantic fusion features; in the second stage, a cross-modal dimension adaptation method based on projection alignment is used to solve the problem of heterogeneous feature dimension mismatch, and a dual-modal gating mechanism is designed to dynamically balance the weights of the graph network topology features and the visual-semantic fusion features, realizing fine-grained dynamic allocation of feature weights.

[0132] The model training is based on the character cluster classification loss L cls The output result of the supervised language constraint based graph network topology feature extraction module is supervised, and a sequence alignment loss L ctc The output result of the overall model is supervised; the overall loss L total The calculation formula is as follows:

[0133]

[0134] L ctc =-log P(y|H fused )

[0135] L total =αL cls +βL ctc

[0136] Wherein, K is the number of character clusters, C is the number of character cluster categories, y k,c is the true label of the kth character cluster, p k,c is the probability that the kth character cluster belongs to category c predicted by the model; y is the true character sequence label, L total represents the total loss of the model, and alpha and beta represent hyperparameters for balancing the loss; P(y|H fused ) represents the probability of predicting the true character sequence y under the condition of given fusion feature sequence H fused .

[0137] Further, the specific steps of Step4 are:

[0138] Step4.1, in order to meet the needs of model batch processing and input normalization, first, the Laos language text image to be recognized is uniformly processed in size, and the specific operation is: the height of the input image is fixed to 32 pixels, and the width is set to W max For images with a width less than W max , the size is padded by filling pixels on the edge to ensure that all input images meet the unified input specification, facilitating batch processing and efficient recognition of the model;

[0139] Step 4.2, save the model weight of the Lao language text image recognition model based on the graph network topology correction which is trained and converged in the previous stage as a ".pth" format model file; then, based on the Sanic framework, a Web service environment is built, the model weight file is loaded into the server memory, and the model is quickly called and online recognition support is realized;

[0140] Step 4.3, relying on the Sanic framework, the development work of the backend service is completed, the key processes of the model are encapsulated, the standard API interface is designed, and the unified calling of the Lao language text image recognition service is realized; at the same time, the service is deployed to the server to support the multi-concurrent access request of the Web end;

[0141] Step 4.4, in the front-end page, the API interface deployed in the server end is called, the Lao language text image to be recognized is input, and the recognition process is triggered; after the server end completes the image recognition and result prediction, the recognition result is returned to the front end and visualized in the page, and finally the complete process verification and function test of the Lao language text image recognition system are realized.

[0142] The application provides a Lao language text image recognition device based on graph network topology correction, which comprises a module for executing the Lao language text image recognition method based on graph network topology correction. Specifically, it can include:

[0143] A Lao language text image dataset construction module: in order to solve the problem of lack of Lao language text image data in a low-resource scenario, a combination of manual collection, algorithm synthesis and data enhancement is used to construct Lao language text images containing various backgrounds, fonts and styles;

[0144] A graph network topology feature extraction module based on language constraints: in order to solve the semantic break problem of traditional segmentation models caused by the topological heterogeneity of Lao language characters, a character topology graph construction method based on linguistic constraints and a global graph reasoning method based on graph convolution network are used to construct and condense the feature map output by the image encoder based on Transformer, so as to extract the graph network topology features;

[0145] A cross-modal feature fusion module based on a gating mechanism: for heterogeneous features such as visual features, text features and graph network topology features, a two-stage gating fusion mechanism is designed; in the first stage, cross-modal attention and gating mechanism are used to fuse visual features and text features; in the second stage, a dimension projection alignment method and a double-modal gating mechanism are used to dynamically balance the weight of the graph network topology features and the visual-semantic fusion features, so as to realize the deep fusion of cross-modal features;

[0146] The Lao text image recognition module is used for encapsulating and deploying the built Lao text image recognition model based on graph network topology correction to the server side, designing a unified API interface, and realizing automatic recognition and result return of the Lao text image.

[0147] In order to verify the effect of the Lao text image recognition model based on graph network topology correction proposed in the application, experiments are performed on 5 million self-constructed Lao text images. About 30,000 of them are manually annotated, and the rest are synthesized by algorithm. The following comparative tests are designed for analysis;

[0148] The Lao language sequence rate precision (Sequence Accuracy, SA) is used as an evaluation index in the experiment, as follows:

[0149]

[0150] Among them, SA, SL and LN represent the sequence accuracy, the total number of correct sequences and the total number of sequences of the Lao text image recognition, respectively.

[0151] Experiment 1: Mainly comparing experimental results;

[0152] Table 1 Main experimental results

[0153]

[0154]

[0155] The Lao language scene image recognition method based on graph network topology correction shows significant performance improvement compared to the traditional fusion language error correction model recognition method based on Transformer. As shown in the table, the GMTL-large model outperforms the MFL-large by 95.7% with an accuracy of 96.4%, and the parameter amount only increases by 4.3%, verifying the high efficiency of the graph topology correction module. The lightweight version GMTL-tiny improves the accuracy by 0.3 percentage points compared to the same level MFL-tiny, and the parameter amount increases by 11.6%, indicating that the module is still applicable in the small model scene.

[0156] Further analysis shows that the GMTL series performs particularly well in complex typesetting text (such as nested superscripts and subscripts). The GMTL-base model improves the accuracy by 0.7 percentage points compared to the MFL-base, and the parameter amount only increases by 11.0%, proving that the graph network effectively alleviates the semantic break problem caused by local feature bias in traditional methods by explicitly modeling the topology relationship of character clusters. The experimental results verify that the topology correction module can significantly improve the robustness of the model to complex Lao text without significantly increasing the computational cost.

[0157] Experiment two: ablation experiment results;

[0158] To verify the effectiveness of the feature reshaping module and the change of the feature fusion position, the application carries out combination test based on GMFL-base. The experimental results are shown in Table 2 (“X” represents not adopted, and “√” represents adopted):

[0159] Table 2 Influence of subgraph construction module on recognition results

[0160]

[0161] As shown in Table 2, the subgraph construction module has a decisive influence on the model performance. When the module is not enabled, the recognition accuracy is 94.2%, and after introducing the module, it is improved to 96.0%, and the performance gain is 1.8 percentage points. This shows that the subgraph construction based on the character cluster topology can effectively model the symbiotic relationship between base characters and superscripts and subscripts, suppress noise connections and strengthen local morphological features, and significantly improve the feature representation quality

[0162] In order to verify the effectiveness of the application for other recognition methods, the method in this chapter is tried to be circumscribed on other baseline experiments to verify its effectiveness, and the experimental results are shown in Table 3 (“X” represents not adopted, and “√” represents adopted).

[0163] Table 3 Ablation experiment on other baseline methods

[0164]

[0165]

[0166] Table 3 further verifies the universality and effectiveness of the graph topology correction module. When the module is circumscribed to different baseline models, it brings stable improvement: the recognition accuracy of ResNet50+BiLSTM+Attention is improved from 87.7% to 89.5%, and ViT-STR is improved from 93.0% to 94.1%. Especially in the method in this chapter, the introduction of the module makes the recognition accuracy optimized from 95.3% to 96.0%, which shows its compatibility with the language correction model. The experimental results prove that the module can make up for the limitations of pure visual or sequence models by explicitly modeling the character cluster topology rules, and provide an extensible feature enhancement scheme for heterogeneous models.

[0167] In order to ensure the authenticity and effectiveness of the experiment, the application additionally labels 1000 real scene images in an artificial labeling manner, and uses them as a test set. The application carries out test experiment on the 1000 real scene test sets, and the experimental results are shown in Table 4.

[0168] Table 4 Experimental results on real test set

[0169]

[0170] As can be seen from Table 4, the method of the present application further improves 0.6 percentage points compared with the fusion language correction model recognition method based on the Transformer on the real scene Lao text image test set, verifying the adaptability of the graph topology correction module to complex text scenes. Compared with the traditional method, the method of the fourth chapter is significantly better than ResNet50+BiLSTM+CTC and ResNet50+Transformer, especially when processing low-quality images, it significantly reduces the misrecognition and omission rate of superscripts and subscripts by explicitly modeling the character cluster topological relationship. Compared with the current optimal iterative method ABINet, the method of the present application improves 1.4 percentage points by fusing the graph structure and cross-modal features under the condition of similar parameter amount, indicating its strong robustness to long-tail characters. The experimental results prove that the graph topology correction module can effectively capture the geometric and semantic constraints of Lao characters, make up for the limitations of pure visual or language models, and provide a better solution for low-resource language text recognition in real scenes.

[0171] The specific embodiments of the present application are described in detail above in combination with the drawings, but the present application is not limited to the above-mentioned embodiments, and various changes can be made within the knowledge possessed by those skilled in the art without departing from the purpose of the present application.

Claims

1. A method for image recognition of Lao text based on graph network topology correction, characterized in that: The method comprises: Step1, Lao text image dataset construction and preprocessing: adopt a combination of manual collection, algorithm synthesis and data enhancement to construct Lao text images containing various backgrounds, fonts and styles; Step2, design a graph network topology feature extraction module based on language constraints: adopt a character topology graph construction method based on linguistic constraints and a global graph reasoning method based on graph convolution network to construct and condense the feature map output by the image encoder based on Transformer, and realize the extraction of the graph network topology feature; Step3, design a cross-modal feature fusion module based on a gating mechanism: for heterogeneous features such as visual features, text features and graph network topology features, design a two-stage gating fusion mechanism; in the first stage, fuse visual features and text features through cross-modal attention and gating mechanism; in the second stage, use dimension projection alignment method and double-mode gating mechanism to dynamically balance the weight of graph network topology feature and visual-semantic fusion feature, and realize deep fusion of cross-modal features; Step4, Lao text image recognition: encapsulate and deploy the Lao text image recognition model based on graph network topology correction constructed in Step2 and Step3 to the server side, design a unified API interface, and realize automatic recognition and result return of Lao text images.

2. The method for recognizing Lao text image based on graph network topology correction according to claim 1, characterized in that: The specific steps of Step1 are: Step1.1, obtain original Lao text images through network crawling, cover multiple environments, and ensure data diversity; at the same time, generate synthetic images based on conditional diffusion model to construct combinations including base characters and attached superscript and subscript symbols; Step1.2, label the collected images in the text area, and label the corresponding Lao text content to ensure character-level alignment; adopt automatic filtering and manual checking mechanism to eliminate low-quality, fuzzy or incorrectly labeled text images; Step1.3, adopt multiple data enhancement strategies to improve data diversity and model robustness; including: scaling: adjust the image size to simulate different resolutions; geometric transformation: rotate, tilt and translate the image; occlusion and cropping: randomly or centrally crop part of the image area; noise injection: add random noise interference; light and contrast adjustment: adjust brightness, contrast and saturation to enhance the adaptability of the image under different lighting conditions; Step1.4, finally construct a large-scale dataset covering various text forms, and divide it into training set, validation set and test set.

3. The method for recognizing Lao text image based on graph network topology correction according to claim 1, characterized in that: In Step2, the graph network topology feature extraction module based on language constraints comprises two parts: feature construction and feature condensation; The feature construction part adopts a character topology graph construction method based on linguistic constraints to construct character clusters through spatial similarity clustering of the feature map output by the image encoder based on Transformer, and then converts linguistic constraints into graph topology structure, generates local subgraphs with geometric center as root node and links them into global graph, realizes the construction of graph network topology feature; The feature condensation part adopts a global graph reasoning method based on graph convolutional networks, which performs hierarchical modeling and information aggregation of local and global structures, and performs feature condensation through stacked graph convolution and graph pooling operations, thereby realizing the extraction of graph network topological features.

4. The method for recognizing Lao text image based on graph network topology correction according to claim 1 or 3, characterized in that: In Step 2, the character topology graph construction method based on linguistic constraints constructs graph network topology features; including: By introducing the concept of Lao character clusters, the morphological co-occurrence relationship between base characters and subscripts and superscripts is explicitly modeled, and linguistic constraints are transformed into graph topology. Here, a character cluster is defined as the smallest semantic unit composed of a base character and its attached subscripts and superscripts. Then, the inductive bias of the graph network is used to capture the spatial-semantic dependencies inside and outside the character cluster, which alleviates feature ambiguity caused by subscript or superscript offset or fuzziness.

5. The method for recognizing Lao text image based on graph network topology correction according to claim 4, characterized in that: In the specific operation, firstly, the character cluster segmentation and root node positioning stage is adopted, the geometric driving clustering algorithm is adopted, the character cluster area is divided based on the spatial density and direction consistency of the character pixels; the geometric center (x c ,y c ) of the character cluster is taken as the root node v root in each cluster, the remaining pixel nodes are divided according to the relative position of the coordinates, the coordinates of the i-th pixel node are (x i ,y i ), the coordinates of the root node v root are (x c ,y c ), and the Euclidean distance d i of the pixel node to the root node is calculated as follows: d i = ||(x i -y j )-(x c ,y c )||2 In the local subgraph construction phase, a weighted subgraph is constructed within the character cluster where the edge weight w in the adjacency matrix E ij It is determined by the node feature similarity and the spatial proximity, and the calculation process is as follows: w ij = σ (cos (h i , h j ) · exp (-γd ij )) wherein h i , h j is the visual feature of the node, cos() is the cosine similarity, γ is the distance attenuation coefficient, σ is the activation function, d ij is the Euclidean distance between node i and node j, used to measure the proximity of the two nodes in the spatial position; Only w ij Strong edges with a correlation of >0.7 to suppress noise interference; after completion of the construction of the local subgraphs The connection of the global subgraph G global = (V, E) is completed by linking the root nodes of the local subgraphs in sequence to the 2-level complete graph.

6. The method for recognizing Lao text image based on graph network topology correction according to claim 1 or 3, characterized in that: In Step 2, the topological features of the graph network are extracted using a global graph inference method based on graph convolutional networks; including: First, the initial global sub-graph G global = (V, E) is aggregated layer by layer using graph convolutional network to aggregate node features, and the calculation process is as follows: where E ij is the edge weight in the adjacency matrix, W (l) is the learnable weight matrix, and σ is the activation function, is the feature vector of node i after graph convolution in the l-th layer, which is the visual feature when l = 0 and the abstract feature when l > 0 by fusing the topological relationship; is the updated feature vector of node i after graph convolution in the (l+1)-th layer. After each convolutional layer, differentiable pooling is used to dynamically compress redundant nodes, and key topological structures are preserved through node importance scoring. The calculation process is as follows: Pooling matrix The number of nodes is reduced from N l to N l+1 , and low-contribution nodes are gradually removed; after L-layer iteration, the features of each character cluster subgraph are compressed to the root node v root , forming a concise graph network topology feature H global ∈R K×d , where K is the number of decoded characters, that is, the total number of character clusters, d is the dimension of each character feature, and the output dimension from the Transformer encoder. This process achieves multi-granularity feature representation through hierarchical abstraction: at the local level, it models the morphological symbiotic relationship between base characters and subscripts and superscripts, and at the global level, it captures the reading order and semantic coherence between character clusters.

7. The method of claim 1, wherein the method is based on a graph network topology correction of a Lao language text image recognition method. Step 3 includes: First, the visual Transformer-based text recognition model extracts the visual features H of the input image vis ∈R N×d where N represents the length of the visual feature sequence, and d represents the dimension of the feature; then the visual feature is sequentially decoded to obtain a preliminary recognition result is input into the encoder of the grammar error correction model GEC, and the text feature H is output text ∈R M×d where M represents the length of the text feature sequence; To integrate visual and textual features, a cross-modal attention mechanism is employed: where Q = H text W Q , K = H vis W K , V = H vis W V , W Q ∈ R d×dk , W K ∈ R d×dk , W V ∈ R d×dk is a learnable attention weight matrix, dkis the key dimension of the attention mechanism. Then, a first-stage gating mechanism is used to generate visual-semantic fusion features: G1 = σ(W g1 [H vis ; H text + b g1 ) H vs = G1 ⊙ H vis + (1 - G1) ⊙ Attention(Q, K, V) where W g1 ∈R 2d×d is the first-stage gating weight matrix, b g1 ∈R d is the first-stage gating bias vector, G1∈R N ×d is the first-stage gating coefficient, used to dynamically balance the weights of the original visual features and the cross-modal attention features; H vs ∈R N×d is the visual-semantic fusion feature, σ is an activation function, and ⊙ represents element-wise multiplication. The generated visual-semantic fusion feature H vs ∈R L×d Adjust to consistent with the graph node sequence dimension by linear projection: H' vs = W p H vs + b p where W p ∈ R N×K is the dimension projection weight matrix, b p ∈ R K is the projection bias vector, H′ vs ∈ R K×d is the aligned visual-semantic fusion feature adjusted by linear projection, and K is the number of character clusters. The second stage gating mechanism is designed to dynamically balance the graph network topology feature H global ∈R K×d The aligned visual-semantic feature H' vs ∈R K×d The contributions of both, the specific calculation formula is shown as follows: G2 = σ(W g2 [ H global ; H' vs ] + b g2 ) H fused = G2 o tanh(W t H global ) + (1 - G2) o H' vs wherein W g2 ∈R 2d×d is a second stage gating weight matrix, b g2 ∈R d is a second stage gating bias vector, G2∈R K×d is a second stage gating coefficient, used to dynamically adjust the fusion weight of the graph network topology feature and the visual-semantic feature; H fused ∈R K×d is the final fusion output feature, W t ∈R d×d is a graph network feature transformation matrix, σ is an activation function, and ⊙ represents element-wise multiplication. Finally, the fusion output feature H fuse is input into a fully connected layer to generate the final recognition result. The cross-modal feature fusion module based on gating mechanism achieves deep fusion of cross-modal features through a two-stage gating design: the first stage uses a gating mechanism to fuse visual and textual features to generate visual-semantic fusion features; the second stage solves the problem of heterogeneous feature dimension mismatch by using a cross-modal dimension adaptation method based on projection alignment, and designs a dual-modal gating mechanism to dynamically balance the weights of the graph network topology features and visual-semantic fusion features, thereby achieving fine-grained dynamic allocation of feature weights. The model is trained with a character cluster classification loss L cls The output result of the language constraint based graph network topology feature extraction module is supervised with a sequence alignment loss L ctc The output result of the whole model is supervised; the whole loss L total The calculation formula is as follows: L ctc = -log P(y | H fused ) L total = aL cls + bL ctc where K is the number of character clusters, C is the number of character cluster classes, y k,c is the true label of the kth character cluster, p k,c is the probability that the kth character cluster predicted by the model belongs to class c; y is the true character sequence label, L total represents the total loss of the model, and a, b represent hyperparameters for balancing the loss, P(y|H fused represents the probability of predicting the true character sequence y under the condition of given the fused feature sequence H fused .

8. The method for recognizing Lao text image based on graph network topology correction according to claim 1, characterized in that: The specific steps of Step 4 are as follows: Step 4.1, first, the Laos language text image to be identified is uniformly sized, and the specific operation is: the height of the input image is fixed to 32 pixels, and the upper limit of the width is set to W max For images with a width less than W max , the size is filled by padding pixels at the edge to ensure that all input images meet the uniform input specification, facilitating batch processing and efficient recognition of the model. Step 4.2: Save the model weights of the Lao text image recognition model based on graph network topology correction that has been trained and converged in the previous stage, and store it as a ".pth" format model file; then, build a web service environment based on the Sanic framework, load the model weight file into the server memory, and realize the fast calling of the model and online recognition support; Step 4.3: Based on the Sanic framework, complete the development of the backend service, encapsulate the key processes of the model, design standard API interfaces, and realize the unified call of Lao text image recognition service; at the same time, deploy the service to the server to support multiple concurrent access requests from the web client. Step 4.4: On the front-end page, by calling the API interface deployed on the server, input the Lao text image to be recognized to trigger the recognition process; After the server completes image recognition and result prediction, it returns the recognition results to the front end and displays them on the page, thus realizing the complete process verification and functional testing of the Lao text image recognition system.

9. The device for recognizing Lao text image based on graph network topology correction, characterized in that, A computer program product including a computer readable medium storing program code, the program code configured to cause a computer to perform the method of any one of claims 1-8. A computer program product including a computer readable medium storing program code, the program code configured to cause a computer to perform the method of any one of claims 1-8.

Images