A text recognition method based on image global information query decoding

Abstract

The present invention discloses a text recognition method based on image global information query decoding. This method introduces global image information as a decoding condition for the decoder, shares it in all time steps, and implements adaptive transformation of the shared global image information through a self-attention mechanism, thereby retrieving the character-level features corresponding to each time step in the cross-attention layer. At the same time, a series of encoding constraint components are proposed to alleviate the problem of inconsistent global image feature quality faced by the constructed text recognizer in multilingual recognition. The present invention has demonstrated effectiveness and versatility in tasks such as multilingual text image recognition (Chinese and English) and multi-scene text image recognition (natural scenes, network text), and has achieved significant performance improvement in irregular text image recognition.

Description

Technical Field

[0001] The invention belongs to the fields of pattern recognition and optical character recognition, and relates to a text recognition method based on image global information query decoding. Background Art

[0002] Optical character recognition (OCR) has long played a vital role in the fields of artificial intelligence and pattern recognition. Scene text recognition (STR) has been a research hotspot in recent years, enabling the conversion of text line images in natural scenes into computer-processable text strings. Attention-based STR methods are currently the main research approach, capturing semantic dependencies by constructing language contexts and retrieving character-level features at the corresponding time step from all text image features. However, these methods commonly suffer from the following two problems:

[0003] 1. During the initial decoding phase, due to insufficient contextual information, the recognition model is prone to errors. We define this phenomenon as the "initial decoding context missing" problem.

[0004] 2. It is extremely susceptible to the irregular characteristics of the character area in the text image, which leads to errors in the character features sampled at local time steps. This phenomenon is usually called the "attention drift" problem.

[0005] To solve the above problems, the present invention designs a non-autoregressive text recognition method NASTR based on image global information query decoding. This method introduces global image information as a decoding condition for the decoder, shares it in all time steps, and realizes adaptive transformation of the shared global image information through the self-attention mechanism, and then retrieves the character-level features corresponding to each time step in the cross-attention layer; at the same time, a series of encoding constraint components are proposed to alleviate the problem of inconsistent global image feature quality faced by the constructed text recognizer in multilingual recognition. The reason for this problem is that the difficulty of Chinese and English text image recognition is different, and extracting high-quality global information of Chinese text images usually requires additional encoding constraints. The recognition model based on image global information query decoding proposed in the present invention has shown effectiveness and versatility in tasks such as multilingual text image recognition (Chinese, English) and multi-scene text image recognition (natural scenes, network text), and has achieved significant improvement in irregular text image recognition performance. Summary of the Invention

[0006] To address the shortcomings of existing text recognition technologies, the present invention provides a text recognition method based on global image information query decoding. This method overcomes the initial context loss problem of previous semantic context query decoding methods, thereby improving attention drift and enhancing the performance of the recognizer for irregular text images. Furthermore, because the global image information provided by the present invention is time-step-independent, parallel decoding is possible, resulting in superior recognition speed compared to context-based recognition models.

[0007] The technical solution of the present invention mainly includes the following steps:

[0008] Step 1: Encode the input text image using an image feature extractor is the corresponding image feature Where H, W are the height and width of the text image, "3" is the number of image channels, d, h, w are the number of channels, height and width of the image features, respectively; further I Perform global average pooling calculation to obtain the global features of the image F I Perform flattening to get the image visual markup Where m = h × w.

[0009] Step 2: Use three pre-defined feature encoding constraint components to transform F into different output forms, and through additional training objectives, ensure that F contains sufficient global image information. The three pre-defined feature encoding constraint components include a bidirectional image-text global alignment module (ITC) for language knowledge constraints, a length predictor (LP) module for global structural constraints, and a disordered character predictor (DCP) module for global semantic constraints.

[0010] Step 3: Copy T copies of F to obtain the image global information query decoding matrix shared by all decoding time steps Where T is the preset maximum decoding time step. q With F I As input, the representation matrix is ​​obtained through a decoding module consisting of a multi-head self-attention layer, a multi-head cross-attention layer, and a forward propagation layer. in Represents the feature representation corresponding to the t-th character.

[0011] Step 4: Map F through a linear classifier shared across T time steps r Output for classification Where C represents the number of preset character categories; cls Identification tag corresponding to X Calculating cross entropy loss where y trepresents the expected character category at the t-th time step.

[0012] Step 5: Calculate the global feature encoding constraint loss for the image oriented language knowledge constraint Image global feature encoding constraint loss for global structure constraints and image global feature encoding constraint loss for global semantic constraints Further optimize the global image information. Combined with Linear combination and Get the final loss and use backpropagation to update the non-autoregressive text recognition NASTR model parameters.

[0013] Step 6: Implement text recognition through the trained non-autoregressive text recognition NASTR model.

[0014] The step 1 specifically includes the following steps:

[0015] 1-1: Using the φ parameterized Vision Transformer (ViT) model as the image feature extractor, the input scene text image Encoded as the corresponding d-dimensional image feature The process can be described as F I =f φ (X).

[0016] 1-2: Perform an average pooling operation along the spatial dimension of the image feature, and pool the feature F with a spatial size of h×w I Transformed into a 1-dimensional vector feature with collapsed spatial information As the global information corresponding to the image, the process can be described as F = AvgPool (F I ).

[0017] 1-3: Perform a flattening operation along the spatial dimension of the image feature to convert the spatial size of the feature F to h×w I Transformed into sequence features As input information for the subsequent multi-head cross attention layer in the decoder.

[0018] Step 2, based on step 1, applies encoding constraints to the global image feature F during the training phase to address the degradation of F representation quality caused by language or scene changes. This invention designs three encoding constraint components, designing corresponding modules from the perspectives of language knowledge, global structure, and global semantics, respectively. This improves the encoding quality of F through supervised optimization. The specific implementation steps are as follows:

[0019] 2-1: Bidirectional image-text global alignment module based on language knowledge constraints:

[0020] 2-1-1: First, the recognition label Y corresponding to the input text image X is converted into [y1,…,y t ,…,y T ], by embedding the matrix Mapped to the corresponding d-dimensional vector representation Where T and C are the preset maximum character sequence length and the preset number of character categories respectively;

[0021] 2-1-2: Then use the Transformer encoder (including multi-head self-attention layer and forward propagation layer) to transform F T Encoded as contextual semantic features And further Perform average pooling along the time dimension to obtain global text information By aligning F l Combined with the global image information F, the encoded F contains the global language information of the text image, thereby enhancing the quality of subsequent query decoding.

[0022] 2-2: Length predictor module for global structural constraints:

[0023] 2-2-1: First, calculate the number of characters corresponding to the input text image X As the supervisory signal for the length predictor module, L contains the actual number of characters and an additional special marker indicating the end of prediction. For example, if the text content in X is "invention patent", then L = 5.

[0024] 2-2-2: Then use a multilayer perceptron (MLP) with one hidden layer to predict the number of characters in the input text image using the global image information F as input. The predicted value is expressed as a probability vector The process can be described as p l =W (2) σ(W (1) (F)), where Both are learnable linear transformation matrices, and σ(·) is the ReLU activation function.

[0025] 2-3: Unordered character predictor module for global semantic constraints:

[0026] 2-3-1: First, calculate the character frequency distribution vector corresponding to the input text image X relative to the preset character category number C And q c satisfy First, perform deduplication on the character content contained in X to obtain the potential character category set S corresponding to X. For example, if the text content in X is "I love invention, invention patent, text recognition, pattern recognition", then S = {I, love, invention, patent, text, character, recognition, pattern}. c It can be calculated by the following formula:

[0027]

[0028] Where i=1,…,C is the index of each character category, and |S| represents the number of elements in the set.

[0029] 2-3-2: Then use a multilayer perceptron (MLP) with one hidden layer, taking the global image information F as input, to predict the probability of each character class in the input text image X The process can be described as p c =W (2) σ(W (1) (F)), where Both are learnable linear transformation matrices, and σ(·) is the ReLU activation function.

[0030] The step 3 is based on the global image information F and image visual mark F obtained in step 1. v , construct the global to local character recognition and decoding process to achieve text image recognition. The specific steps are as follows:

[0031] 3-1: Copy the global image feature F obtained in step 1-2 to T copies, and get Let it be the initial input of T decoding time steps, the process can be described as F q =F.unsqueeze(dim=0).repeat(T,1).

[0032] 3-2: Create a position encoding matrix and compare it with F q Superposition. Use sine and cosine fixed position coding to construct the position coding matrix With F q After adding, we get Serves as input to the complete decoder.

[0033] 3-3: Transformation through Multi Head Self-Attention (MHSA) is the time-step specific decoding feature, the process can be described as Where LN represents the layer normalization operation.

[0034] 3-4: Through the Multi Head Cross-Attention (MHCA) layer, using F d From the image visual mark F v Retrieve character-level feature representations from the dataset and map them into representation matrices through the feed forward network (FFN) layer. The process can be described as F r =FFN(LN(MHCA(F d ,F v )+F d )).

[0035] In step 5, based on step 2, the losses of the three encoding constraints are calculated respectively to further optimize the global image information F. Then, combined with the loss results in step 4, the preset hyperparameter λ is used to calculate the loss of the three encoding constraints. ITC ,λ LP ,λ DCP Linearly combine the four loss values ​​to get the total loss, and then use the backpropagation algorithm to optimize the entire model. The specific implementation steps are as follows:

[0036] 5-1 calculates the losses of the three encoding constraints respectively to further optimize the global image information. The process can be described as:

[0037] 5-1-1: Bidirectional Image-Text Global Alignment Loss

[0038] For a training batch with N text images and N text annotations Using the process described in steps 1 and 2, we get the text image X i Corresponding global image features and corresponds to the text label Y i Global language features From this we can calculate in, and They are the image-to-text and text-to-image alignment losses, respectively, and are calculated using the following formulas:

[0039]

[0040] in and The global language features and global image features in the batch are enumerated respectively, and They represent the dot product of the i-th global image feature and the i-th global language feature, respectively, expressing the similarity between the two. The superscripts I and T distinguish the specific directions of the bidirectional alignment calculation. τ is the preset temperature parameter.

[0041] 5-1-2: Length prediction loss

[0042] As described in step 2-2, length prediction is modeled as a classification problem with T number of categories, so the length prediction loss is Using cross entropy, we can label the number of characters L and the probability vector p output by the length predictor. l For input,

[0043] 5-1-3: Out-of-order character prediction loss

[0044] Take the character frequency distribution q obtained in step 2-3 c and the probability vector p output by the out-of-order character predictor c As input, the unordered character prediction process is modeled as a distribution alignment problem. It is achieved through KL divergence (Kullback-Leibler Divergence). The specific calculation process is:

[0045] Using only the average pooling of features output by the feature extractor as global image information, without imposing any additional constraints, F faces the problem of high similarity across different text images (e.g., scene changes, language changes), which weakens the guidance of the text-specific decoding process. The three encoding constraint components designed in step 2 can meet the hierarchical requirements of global image features in multi-language and multi-scene variations, enhancing the input text image-specific information in the global image features. This enhancement improves the distinguishability between global image features from different text images.

[0046] 5-2: Through the preset hyperparameter λ ITC ,λ LP ,λ DCP Linearly combine the four loss values ​​and calculate the total loss. The process can be described as:

[0047]

[0048] 5-3: Use backpropagation to update the parameters of the non-autoregressive text recognition NASTR model.

[0049] The non-autoregressive text recognition NASTR model is composed of the image feature extractor, feature encoding constraint component, decoding module and linear classifier described in steps 1 to 4.

[0050] The beneficial effects of the present invention are as follows:

[0051] In response to the problems of missing initial decoding context semantics and attention drift in existing scene text recognition technologies, the present invention proposes a text recognition method based on image global information query decoding: by introducing global image information, high-quality decoding query clues are maintained at all decoding time steps. Three encoding constraint components are proposed to solve or enhance the global language knowledge, global structural knowledge and global semantic context knowledge in the global image information. The technology proposed in the present invention not only overcomes the above-mentioned defects, but also can output decoding predictions for all time steps in parallel during the decoding stage, achieving the dual advantages of speed and accuracy. From an algorithmic point of view, it is a new scene text recognition architecture; from an engineering point of view, the parallel processing advantages of the technology proposed in the present invention are easy to deploy and can be applied to actual engineering projects, giving full play to the value of scene text recognition algorithms in various practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is a schematic diagram of the overall process of the text recognition method based on image global information query decoding proposed by the present invention.

[0053] Figure 2 This is an example of some recognition results of the present invention on the scene text recognition dataset and the time-step decoding attention visualization. DETAILED DESCRIPTION

[0054] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. The following description is only for demonstration and explanation and does not limit the present invention in any form.

[0055] The general identification steps have been described in detail in the "Summary of the Invention". Now, the identification process of the present invention is specifically described with reference to examples. Figure 1 It is a schematic diagram of the overall processing flow.

[0056] The technical solution of the present invention is mainly divided into the following steps:

[0057] Step 1: Input the text image into the non-autoregressive text recognition NASTR model;

[0058] The non-autoregressive text recognition NASTR model first encodes the text image using an image feature extractor is the corresponding image feature Where H, W are the height and width of the text image respectively, "3" is the number of image channels, d, h, w are the number of channels, height and width of the image features respectively; further I Perform global average pooling calculation to obtain the global features of the image F IPerform flattening to get the image visual markup Where m = h × w. The proposed NASTR is evaluated using Chinese and English STR datasets. For Chinese, training and evaluation are performed on a large-scale Chinese text recognition dataset. This dataset contains 509,164 scene dataset training samples, 63,645 validation samples, and 63,646 test samples. The Web dataset contains 112,471 training, 14,059 validation, and 14,059 test samples, respectively. For English text recognition, the proposed method is trained on the training set of the recently released benchmark Union14ML and the performance of NASTR is evaluated on 6 standard benchmarks and the corresponding Union14M-Benchmark test dataset.

[0059] Step 1 specifically includes the following steps:

[0060] 1-1: Using the φ parameterized Vision Transformer (ViT) model as the image feature extractor, the input scene text image Encoded as the corresponding d-dimensional image feature Where d is set to 384 and patch is set to 4×4. The process can be described as F I =f φ (X).

[0061] 1-2: Perform an average pooling operation along the spatial dimension of the image feature, and pool the feature F with a spatial size of h×w I Transformed into a 1-dimensional vector feature with collapsed spatial information Where d is set to 384 as the global information corresponding to the image. The process can be described as F = AvgPool (F I ).

[0062] 1-3: Perform a flattening operation along the spatial dimension of the image feature to convert the spatial size of the feature F to h×w I Transformed into sequence features As input information for the subsequent multi-head cross attention layer in the decoder.

[0063] For Chinese recognition, all text images are resized to 32×256, and no data augmentation is performed for fair comparison with previous methods; for English recognition, all text images are resized to 32×256, and data augmentation effects such as rotation and perspective distortion are performed.

[0064] Step 2: Use three pre-defined feature encoding constraint components to transform F into different output forms, and through additional training objectives, ensure that F contains sufficient global image information. The three pre-defined feature encoding constraint components include a bidirectional image-text global alignment module (ITC) for language knowledge constraints, a length predictor module (LP) for global structural constraints, and a disordered character predictor module (DCP) for global semantic constraints.

[0065] Step 2, based on step 1, applies encoding constraints to the global image features F during the training phase to address the degradation of F representation quality caused by language or scene changes. This paper designs three encoding constraint components, designing corresponding modules from the perspectives of language knowledge, global structure, and global semantics, respectively. This improves the encoding quality of F through supervised optimization. The specific implementation steps are as follows:

[0066] 2-1: Bidirectional image-text global alignment module based on language knowledge constraints:

[0067] 2-1-1: First, the recognition label Y corresponding to the input text image X is converted into [y1,…,y t ,…,y T ], by embedding the matrix Mapped to the corresponding d-dimensional vector representation Where T and C are the preset maximum character sequence length and the preset number of character categories respectively;

[0068] 2-1-2: Then use the Transformer encoder (including multi-head self-attention layer and forward propagation layer) to transform F T Encoded as contextual semantic features And further Perform average pooling along the time dimension to obtain global text information By aligning F l Combined with the global image information F, the encoded F contains the global language information of the text image, thereby enhancing the quality of subsequent query decoding.

[0069] 2-2: Length predictor module for global structural constraints:

[0070] 2-2-1: First, calculate the number of characters corresponding to the input text image X As the supervisory signal for the length predictor module, L contains the actual number of characters and an additional special marker indicating the end of prediction. For example, if the text content in X is "invention patent", then L = 5.

[0071] 2-2-2: Then use a multilayer perceptron (MLP) with one hidden layer to predict the number of characters in the input text image using the global image information F as input. The predicted value is expressed as a probability vector The process can be described as p l =W (2) σ(W (1) (F)), where Both are learnable linear transformation matrices, and σ(·) is the ReLU activation function.

[0072] 2-3: Unordered character predictor module for global semantic constraints:

[0073] 2-3-1: First, calculate the character frequency distribution vector corresponding to the input text image X relative to the preset character category number C And q c satisfy First, perform deduplication on the character content contained in X to obtain the potential character category set S corresponding to X. For example, if the text content in X is "I love invention, invention patent, text recognition, pattern recognition", then S = {I, love, invention, patent, text, character, recognition, pattern}. c It can be calculated by the following formula

[0074]

[0075] Where i=1,…,C is the index of each character category, and |S| represents the number of elements in the set.

[0076] 2-3-2: Then use a multilayer perceptron (MLP) with one hidden layer, taking the global image information F as input, to predict the probability of each character class in the input text image X The process can be described as p c =W (2) σ(W (1) (F)), where Both are learnable linear transformation matrices, and σ(·) is the ReLU activation function.

[0077] Step 3: Copy T copies of F to obtain the image global information query decoding matrix shared by all decoding time steps Where T is the preset maximum decoding time step. q With F I As input, the representation matrix is ​​obtained through a decoding module consisting of a multi-head self-attention layer, a multi-head cross-attention layer, and a forward propagation layer. in Represents the feature representation corresponding to the t-th character.

[0078] Step 3 is based on the global image information F and image visual label F obtained in step 1. v , construct the global to local character recognition and decoding process to achieve text image recognition. The specific steps are as follows:

[0079] 3-1: Copy the global image feature F obtained in step 1-2 to T copies, and get Let it be the initial input of T decoding time steps, the process can be described as F q =F.unsqueeze(dim=0).repeat(T,1).

[0080] 3-2: Create a position encoding matrix and compare it with F q Superposition. Use sine and cosine fixed position coding to construct the position coding matrix With F q After adding, we get Serves as input to the complete decoder.

[0081] 3-3: Transformation through Multi Head Self-Attention (MHSA) is the time-step specific decoding feature, the process can be described as Where LN represents the layer normalization operation.

[0082] 3-4: Through the Multi Head Cross-Attention (MHCA) layer, using F d From the image visual mark F v Retrieve character-level feature representations from the dataset and map them into representation matrices through the feed forward network (FFN) layer. The process can be described as F r =FFN(LN(MHCA(F d ,F v )+F d )).

[0083] Step 4: Map F through a linear classifier shared across T time steps r Output for classification Where C represents the number of preset character categories; cls Identification tag corresponding to X Calculating cross entropy loss where y t represents the expected character category at the t-th time step.

[0084] Step 5: Calculate the global feature encoding constraint loss for the image oriented language knowledge constraint Image global feature encoding constraint loss for global structure constraints and image global feature encoding constraint loss for global semantic constraints Further optimize the global image information. Combined with Linear combination and Get the final loss and use backpropagation to update the non-autoregressive text recognition NASTR model parameters.

[0085] Based on step 2, the losses of the three encoding constraints are calculated separately to further optimize the global image information F. Then, combined with the loss results in step 4, the preset hyperparameter λ is used to ITC ,λ LP ,λ DCP Linearly combine the four loss values ​​to get the total loss, and then use the backpropagation algorithm to optimize the entire model. The specific implementation steps are as follows:

[0086] 5-1 calculates the losses of the three encoding constraints respectively to further optimize the global image information. The process can be described as:

[0087] 5-1-1: Bidirectional Image-Text Global Alignment Loss

[0088] For a training batch with N text images and N text annotations In the present invention, i is set to 64. Using the process described in steps 1 and 2, the text image X is obtained. i Corresponding global image features and corresponds to the text label Y i Global language features From this we can calculate in, and They are the image-to-text and text-to-image alignment losses, respectively, and are calculated using the following formulas:

[0089]

[0090] in and The global language features and global image features in the batch are enumerated respectively, and denote the dot product of the i-th global image feature and the i-th global language feature, respectively, expressing the similarity between the two. The superscripts I and T distinguish the specific directions of the bidirectional alignment calculation. τ is a preset temperature parameter, set to 0.07.

[0091] 5-1-2: Length prediction loss

[0092] As described in step 2-2, length prediction is modeled as a classification problem with T number of categories, so the length prediction loss is Using cross entropy, we can label the number of characters L and the probability vector p output by the length predictor. l For input,

[0093] 5-1-3: Out-of-order character prediction loss

[0094] Take the character frequency distribution q obtained in step 2-3 c and the probability vector p output by the out-of-order character predictor c As input, the unordered character prediction process is modeled as a distribution alignment problem. It is achieved through KL divergence (Kullback-Leibler Divergence). The specific calculation process is:

[0095] Using only the average pooling of features output by the feature extractor as global image information, without imposing any additional constraints, F faces the problem of high similarity across different text images (e.g., scene changes, language changes), which weakens the guidance of the text-specific decoding process. The three encoding constraint components designed in step 2 can meet the hierarchical requirements of global image features in multi-language and multi-scene variations, enhancing the input text image-specific information in the global image features. This enhancement improves the distinguishability between global image features from different text images.

[0096] 5-2: Through the preset hyperparameter λ ITC ,λ LP ,λ DCP Linearly combine the four loss values ​​and calculate the total loss. The process can be described as:

[0097]

[0098] For Scene datasets, set to λ LP =0.1.

[0099] For the Web dataset, only the ITC constraint component is used for training, which is set to λ CE =λITC =1.

[0100] For the English dataset, only the recognition loss is required, which is set to λ CE =1.

[0101] In addition, due to the different dataset sizes, the training epochs for Scene and Web datasets are set to 100 and 300, respectively.

[0102] 5-3: Using back propagation to update the parameters of the non-autoregressive text recognition NASTR model. The non-autoregressive text recognition NASTR model is composed of the image feature extractor, feature encoding constraint component, decoding module and linear classifier described in steps 1 to 4.

[0103] Step 6: Implement text recognition using the trained non-autoregressive text recognition (NASTR) model. NASTR achieves recognition accuracy rates of 78.10% and 69.92% on the Chinese Scene and Web datasets, respectively.

[0104] Ablation studies of different combinations of global image feature encoding constraints on Chinese datasets. Here, DCP, LP, and ITC represent the disordered character predictor module for global semantic constraints, the length predictor module for global structural constraints, and the bidirectional image-text global alignment module for linguistic knowledge constraints, respectively.

[0105] DCP LP ITC Scene Web √ √ √ 77.77 69.49 × √ √ 76.20 68.48 × × √ 77.23 69.75 × × × 76.68 68.64

[0106] The impact of different numbers of text encoder stacking layers on the accuracy on Chinese datasets. Here, based on the experimental results of the ablation study of different combinations of global image feature encoding constraints, for the Scene dataset, the LP and DCP components are included in the training, while for the Web dataset, these components are excluded.

[0107]

[0108] The above description is a further detailed description of the present invention in conjunction with specific / preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. Those skilled in the art of the present invention may make various substitutions or modifications to the described embodiments without departing from the scope of the present invention, and such substitutions or modifications should be considered to fall within the scope of protection of the present invention.

[0109] Parts of the present invention that are not described in detail belong to the common knowledge of those skilled in the art.

Claims

1. A text recognition method based on image global information query decoding, characterized in that: The steps include: Step 1: Encode the input text image using an image feature extractor is the corresponding image feature Where H, W are the height and width of the text image, "3" is the number of image channels, d, h, w are the number of channels, height and width of the image features, respectively; further I Perform global average pooling calculation to obtain the global features of the image F I Perform flattening to get the image visual markup Where m = h × w; Step 2: transforming F into different output forms using three preset feature encoding constraint components, and ensuring that F contains sufficient global image information through additional training objectives; the three preset feature encoding constraint components include a bidirectional image-text global alignment module for language knowledge constraints, a length predictor module for global structure constraints, and a disordered character predictor module for global semantic constraints; Step 3: Copy T copies of F to obtain the image global information query decoding matrix shared by all decoding time steps Where T is the preset maximum decoding time step; F q With F v As input, the representation matrix is ​​obtained through a decoding module consisting of a multi-head self-attention layer, a multi-head cross-attention layer, and a forward propagation layer. in Indicates the feature representation corresponding to the t-th character; Step 4: Map F through a linear classifier shared across T time steps r Output for classification Where C represents the number of preset character categories; cls Identification tag corresponding to X Calculating cross entropy loss where y t represents the expected character category at the t-th time step; Step 5: Calculate the global feature encoding constraint loss for the image oriented language knowledge constraint Image global feature encoding constraint loss for global structure constraints and image global feature encoding constraint loss for global semantic constraints Further optimize the global image information; then combine Linear combination and Obtain the final loss and use back propagation to update the parameters of the non-autoregressive text recognition NASTR model; the non-autoregressive text recognition NASTR model is composed of the image feature extractor, feature encoding constraint component, decoding module and linear classifier described in steps 1 to 4; Step 6: Implement text recognition through the trained non-autoregressive text recognition NASTR model.

2. The text recognition method based on image global information query decoding according to claim 1, characterized in that: The step 1 specifically includes the following steps: 1-1: Using the φ parameterized visual Transformer model as the image feature extractor, the input scene text image Encoded as the corresponding d-dimensional image feature The process can be described as F I =f φ (X); 1-2: Perform an average pooling operation along the spatial dimension of the image feature, and pool the feature F with a spatial size of h×w I Transformed into a 1-dimensional vector feature with collapsed spatial information As the global information corresponding to the image, the process can be described as F = AvgPool (F I ); 1-3: Perform a flattening operation along the spatial dimension of the image feature to convert the spatial size of the feature F to h×w I Transformed into sequence features As input information for the subsequent multi-head cross attention layer in the decoder.

3. The text recognition method based on image global information query decoding according to claim 2, characterized in that: In step 2, based on step 1, encoding constraints are imposed on the global image feature F during the training phase to address the problem of F representation quality degradation caused by language or scene changes. Three encoding constraint components are designed, with corresponding modules designed from the perspectives of language knowledge, global structure, and global semantics. The encoding quality of F is improved through supervised optimization. The specific implementation steps are as follows: 2-1: Bidirectional image-text global alignment module based on language knowledge constraints: 2-1-1: First, the recognition label Y corresponding to the input text image X is converted into [y1,…,y t ,…,y T ], by embedding the matrix Mapped to the corresponding d-dimensional vector representation Where T and C are the preset maximum character sequence length and the preset number of character categories respectively; 2-1-2: Then use the Transformer encoder to transform F T Encoded as contextual semantic features And further Perform average pooling along the time dimension to obtain global text information By aligning F l Combined with the global image information F, the encoded F contains the global language information of the text image, thereby enhancing the quality of subsequent query decoding; 2-2: Length predictor module for global structural constraints: 2-2-1: First, calculate the number of characters corresponding to the input text image X As a supervisory signal for the length predictor module, where L contains the actual number of characters and an additional special token indicating the end of the prediction; 2-2-2: Then use a multilayer perceptron with one hidden layer to predict the number of characters in the input text image using the global image information F as input; the predicted value is expressed as a probability vector The process can be described as p l =W (2) σ(W (1) (F)), where Both are learnable linear transformation matrices, and σ(·) is the ReLU activation function; 2-3: Unordered character predictor module for global semantic constraints: 2-3-1: First, calculate the character frequency distribution vector corresponding to the input text image X relative to the preset character category number C And q c satisfy First, perform deduplication processing on the character content contained in X to obtain the potential character category set S corresponding to X; further, q c It can be calculated by the following formula: Where i'=1,…,C is the index of each character category, |S| represents the number of elements in the set; 2-3-2: Then use a multi-layer perceptron MLP with one hidden layer, take the global image information F as input, and predict the existence probability of each character class in the input text image X The process can be described as p c =W (4) σ(W (3) (F)), where Both are learnable linear transformation matrices, and σ(·) is the ReLU activation function.

4. A text recognition method based on image global information query decoding according to claim 2 or 3, characterized in that: The specific steps of step 3 are as follows: 3-1: Copy the global image feature F obtained in step 1-2 to T copies, and get Let it be the initial input of T decoding time steps, the process can be described as F q =F.unsqueeze(dim=0).repeat(T,1); 3-2: Create a position encoding matrix and compare it with F q Superposition; Use sine and cosine fixed position coding to construct position coding matrix With F q After adding, we get As a complete decoder input; 3-3: Transformation through Multi-Head Self-Attention Layer MHSA is the time-step specific decoding feature, the process can be described as Where LN represents the layer normalization operation; 3-4: Through the multi-head cross attention layer MHCA, using F d From the image visual mark F v Retrieve the character-level feature representation from the dataset and map it to the representation matrix F through the forward propagation layer FFN r = The process can be described as F r =FFN(LN(MHCA(F d ,F v )+F d )).

5. The text recognition method based on image global information query decoding according to claim 4 is characterized in that: The specific implementation steps of step 5 are as follows: 5-1 calculates the losses of the three encoding constraints respectively to further optimize the global image information. The process can be described as: 5-1-1: Bidirectional Image-Text Global Alignment Loss For a training batch with N text images and N text annotations Using the process described in steps 1 and 2, we get the text image X i Corresponding global image features and corresponds to the text label Y i Global language features From this we can calculate in, and They are the image-to-text and text-to-image alignment losses, respectively, and are calculated using the following formulas: in and The global language features and global image features in the batch are enumerated respectively, and They represent the dot product of the i-th global image feature and the i-th global language feature, respectively, expressing the similarity between the two. The superscripts I and T distinguish the specific directions of the bidirectional alignment calculation; τ is the preset temperature parameter; 5-1-2: Length prediction loss Length prediction loss Using cross entropy, we can label the number of characters L and the probability vector p output by the length predictor. l For input, 5-1-3: Out-of-order character prediction loss Character frequency distribution q c and the probability vector p output by the out-of-order character predictor c As input, the unordered character prediction process is modeled as a distribution alignment problem. It is achieved through KL divergence. The specific calculation process is: 5-2: Through the preset hyperparameter λ ITC ,λ LP ,λ DCP Linearly combine the four loss values ​​and calculate the total loss. The process can be described as: 5-3: Use backpropagation to update the parameters of the non-autoregressive text recognition NASTR model.

Images