Text recognition method based on image global information query decoding

Abstract

The invention discloses a text recognition method based on image global information query decoding. According to the method, global image information is introduced as a decoding condition of a decoder and is shared in all time steps, adaptive transformation of the shared global image information is realized through a self-attention mechanism, and then character-level features corresponding to each time step are retrieved in a cross attention layer; meanwhile, a series of coding constraint components are provided, and the problem that global image feature quality is inconsistent in multi-language recognition of a constructed text recognizer is solved. According to the method, effectiveness and universality are displayed in tasks such as multi-language text image recognition (Chinese and English) and multi-scene text image recognition (natural scenes and web texts), and remarkable irregular text image recognition performance improvement is achieved.

Description

Technical Field

[0001] The invention belongs to the field of pattern recognition and optical character recognition, and relates to a text recognition method based on image global information query decoding. Background Art

[0002] Optical Character Recognition (OCR) has always played an important role in the fields of artificial intelligence and pattern recognition. Among them, Scene Text Recognition (STR) has been a research hotspot in recent years, which realizes the conversion of text line images in natural scenes into computer-processable text strings. The attention-based STR method is the current main research idea. It captures semantic dependencies by building language context and realizes the function of retrieving character-level features of corresponding time steps in all text image features. However, these methods generally have the following two problems:

[0003] 1. In the initial decoding stage, due to the lack of context information, the recognition model is prone to errors. We define this phenomenon as the "initial decoding context missing" problem.

[0004] 2. It is very susceptible to the irregular characteristics of the character area in the text image, which leads to errors in the character features sampled at the local time step. This phenomenon is usually called the "attention drift" problem.

[0005] In order to solve the above problems, the present invention designs a non-autoregressive text recognition method NASTR based on image global information query decoding. This method introduces global image information as the decoding condition of the decoder, shares it in all time steps, and realizes adaptive transformation of the shared global image information through the self-attention mechanism, and then retrieves the character-level features corresponding to each time step in the cross-attention layer; at the same time, a series of encoding constraint components are proposed to alleviate the problem of inconsistent global image feature quality faced by the constructed text recognizer in multilingual recognition. The reason for this problem is that the difficulty of Chinese and English text image recognition is different, and extracting high-quality Chinese text image global information usually requires additional encoding constraints. The recognition model based on image global information query decoding proposed in the present invention shows effectiveness and versatility in tasks such as multilingual text image recognition (Chinese, English) and multi-scene text image recognition (natural scenes, network text), and has achieved significant improvement in irregular text image recognition performance. Summary of the invention

[0006] In view of the shortcomings of the above-mentioned existing text recognition technology, the present invention provides a text recognition method based on image global information query decoding. It makes up for the problem of missing initial context in the previous method based on semantic context query decoding, thereby improving the attention drift problem and enhancing the performance of the recognizer in recognizing irregular text images. In addition, since the image global information provided by the present invention is time-step independent, parallel decoding can be achieved, showing a superior recognition speed compared to the context-based recognition model.

[0007] The technical solution of the present invention mainly includes the following steps:

[0008] Step 1: Encode the input text image using an image feature extractor is the corresponding image feature Where H, W are the height and width of the text image, "3" is the number of image channels, d, h, w are the number of channels, height and width of the image features, respectively; further, F I Perform global average pooling calculation to obtain the global features of the image F I Perform a flattening operation to obtain the image visual markup Where m = h × w.

[0009] Step 2: Use three preset feature encoding constraint components to transform F into different output forms, and make F contain sufficient global image information through additional training objectives. The three preset feature encoding constraint components include a bidirectional image-text global alignment module (ITC) for language knowledge constraints, a length predictor (LP) module for global structure constraints, and a disordered character predictor (DCP) module for global semantic constraints.

[0010] Step 3: Copy T copies of F to obtain the image global information query decoding matrix shared by all decoding time steps Where T is the preset maximum decoding time step. q With F I As input, through the decoding module consisting of a multi-head self-attention layer, a multi-head cross-attention layer, and a forward propagation layer, the representation matrix is ​​obtained in Represents the feature representation corresponding to the t-th character.

[0011] Step 4: Map F through a linear classifier shared across T time steps r Output for classification Where C represents the number of preset character categories; cls The identification tag corresponding to X Calculating cross entropy loss where y trepresents the expected character category at the t-th time step.

[0012] Step 5: By calculating the image global feature encoding constraint loss oriented to the language knowledge constraint Image global feature encoding constraint loss for global structure constraints and image global feature encoding constraint loss for global semantic constraints Further optimize the global image information. Combined with Linear combination and Get the final loss and use back propagation to update the parameters of the non-autoregressive text recognition NASTR model.

[0013] Step 6: Implement text recognition through the trained non-autoregressive text recognition NASTR model.

[0014] The step 1 specifically includes the following steps:

[0015] 1-1: Using the φ-parameterized Vision Transformer (ViT) model as the image feature extractor, the input scene text image Encoded as the corresponding d-dimensional image feature This process can be described as F I =f φ (X).

[0016] 1-2: Perform an average pooling operation along the spatial dimension of the image feature, and pool the feature F with a spatial size of h×w I Transformed into a 1-dimensional vector feature with collapsed spatial information As the global information corresponding to the image, the process can be described as F = AvgPool (F I ).

[0017] 1-3: Flatten the image features along their spatial dimensions to create h×w features F I Transformed into sequence features As input information for the multi-head cross attention layer in the subsequent decoder.

[0018] The step 2 is to impose coding constraints on the global feature F of the image during the training phase on the basis of step 1, so as to solve the problem of quality degradation of F representation caused by language or scene changes. The present invention designs three coding constraint components, and designs corresponding modules from the perspective of language knowledge, global structure, and global semantics, respectively, to improve the coding quality of F through supervised optimization, and the specific implementation steps are as follows:

[0019] 2-1: Bidirectional image-text global alignment module based on language knowledge constraints:

[0020] 2-1-1: First, the recognition label Y corresponding to the input text image X is Y = [y1,…,y t ,…,y T ], by embedding the matrix Mapped to the corresponding d-dimensional vector representation Where T and C are the preset maximum character sequence length and the preset number of character categories respectively;

[0021] 2-1-2: Then use the Transformer encoder (including multi-head self-attention layer and forward propagation layer) to transform F T Encoded as contextual semantic features And further Perform average pooling along the time dimension to obtain global text information By aligning F l Combined with the global image information F, the encoded F contains the global language information of the text image, thereby enhancing the quality of subsequent query decoding.

[0022] 2-2: Length predictor module for global structural constraints:

[0023] 2-2-1: First, calculate the number of characters corresponding to the input text image X As the supervisory signal for the length predictor module, L contains the actual number of characters and an additional special marker indicating the end of prediction. For example, if the text content in X is "invention patent", then L = 5.

[0024] 2-2-2: Then use a multilayer perceptron (MLP) with one hidden layer to predict the number of characters in the input text image using the global image information F as input. The predicted value is expressed as a probability vector The process can be described as p l =W (2) σ(W (1) (F)), where Both are learnable linear transformation matrices, and σ(·) is the ReLU activation function.

[0025] 2-3: Unordered character predictor module for global semantic constraints:

[0026] 2-3-1: First, calculate the character frequency distribution vector corresponding to the input text image X relative to the preset character category number C And q c satisfy First, perform deduplication processing on the character content contained in X to obtain the potential character category set S corresponding to X. For example, if the text content in X is "I love invention, invention patent, text recognition, pattern recognition", then S = {I, love, invention, patent, text, character, recognition, pattern}. c It can be calculated by the following formula:

[0027]

[0028] Among them, i=1,…,C is the index of each character category, and |S| represents the number of elements in the set.

[0029] 2-3-2: Then use a multilayer perceptron (MLP) with one hidden layer, take the global image information F as input, and predict the existence probability of each character class in the input text image X This process can be described as p c =W (2) σ(W (1) (F)), where Both are learnable linear transformation matrices, and σ(·) is the ReLU activation function.

[0030] The step 3 is based on the global image information F and image visual mark F obtained in step 1. v , construct the global to local character recognition decoding process to achieve text image recognition. The specific steps are as follows:

[0031] 3-1: Copy T copies of the image global feature F obtained in step 1-2 to obtain Let it be the initial input of T decoding time steps, the process can be described as F q =F.unsqueeze(dim=0).repeat(T,1).

[0032] 3-2: Create a position encoding matrix and compare it with F q Superposition. Use sine and cosine fixed position coding to construct the position coding matrix With F q After adding, we get As input to the complete decoder.

[0033] 3-3: Transformation through Multi Head Self-Attention (MHSA) is the time-step specific decoding feature, and the process can be described as Where LN represents the layer normalization operation.

[0034] 3-4: Through the Multi Head Cross-Attention (MHCA) layer, using F d From the image visual mark F v Retrieve character-level feature representations from the dataset and map them into representation matrices through the feed forward network (FFN) This process can be described as F r =FFN(LN(MHCA(F d ,F v )+F d )).

[0035] In step 5, based on step 2, the losses of the three encoding constraints are calculated respectively to further optimize the global image information F. Then, combined with the loss result in step 4, the preset hyperparameter λ is used to calculate the loss of the three encoding constraints. ITC ,λ LP ,λ DCP Linearly combine the four loss values ​​to get the total loss, and then use the back propagation algorithm to optimize the entire model. The specific implementation steps are as follows:

[0036] 5-1 calculates the losses of the three encoding constraints respectively to further optimize the global image information. The process can be described as:

[0037] 5-1-1: Bidirectional Image-Text Global Alignment Loss

[0038] For a training batch with N text images and N text annotations Using the process described in steps 1 and 2, we get the text image X i Corresponding global image features and corresponds to the text label Y i Global language features From this we can calculate in, and They are the alignment losses from image to text and text to image, respectively, and are calculated by the following formulas:

[0039]

[0040] in and The global language features and global image features in the batch are enumerated respectively. and They represent the dot product of the ith global image feature and the ith global language feature, respectively, expressing the similarity between the two. The superscripts I and T distinguish the specific directions of the bidirectional alignment calculation. τ is the preset temperature parameter.

[0041] 5-1-2: Length prediction loss

[0042] As described in step 2-2, length prediction is modeled as a classification problem with T number of categories, so the length prediction loss is Using cross entropy implementation, the probability vector p output by the number of characters L and the length predictor is annotated l For input,

[0043] 5-1-3: Out-of-order character prediction loss

[0044] Take the character frequency distribution q obtained in step 2-3 c and the probability vector p output by the out-of-order character predictor c As input, the unordered character prediction process is modeled as a distribution alignment problem. It is realized through KL divergence (Kullback-LeiblerDivergence). The specific calculation process is:

[0045] Only the average pooling of the features output by the feature extractor is used as the global image information. Without any additional constraints, F faces the problem of being highly similar in different text images (such as scene changes, language changes), which weakens the guidance of the text-specific decoding process. The above three encoding constraint components designed in step 2 can meet the hierarchical requirements of global image features to accommodate information under multi-language and multi-scene changes, and enhance the specific information of the input text image in the global image features. This enhancement improves the distinguishability between global image features from different text images.

[0046] 5-2: By preset hyperparameter λ ITC ,λ LP ,λ DCP Linearly combine the four loss values ​​and calculate the total loss. The process can be described as:

[0047]

[0048] 5-3: Use backpropagation to update the parameters of the non-autoregressive text recognition NASTR model.

[0049] The non-autoregressive text recognition NASTR model is composed of the image feature extractor, feature encoding constraint component, decoding module and linear classifier described in steps 1 to 4.

[0050] The beneficial effects of the present invention are as follows:

[0051] In view of the problems of missing initial decoding context semantics and attention drift in existing scene text recognition technologies, the present invention proposes a text recognition method based on image global information query decoding: by introducing global image information, high-quality decoding query clues are maintained at all decoding time steps. And three encoding constraint components are proposed to solve or enhance the global language knowledge, global structural knowledge and global semantic context knowledge in the global image information. The technology proposed in the present invention not only overcomes the above-mentioned defects, but also can output decoding predictions of all time steps in parallel during the decoding stage, achieving the dual advantages of speed and accuracy. From an algorithmic point of view, it is a new scene text recognition architecture; from an engineering point of view, the parallel processing advantages of the technology proposed in the present invention are easy to deploy, can be applied to engineering practice, and give full play to the value of scene text recognition algorithms in various practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 The figure is a schematic diagram of the overall process of the text recognition method based on image global information query decoding proposed by the present invention.

[0053] Figure 2 This is an example of some recognition results of the present invention on the scene text recognition dataset and the time-step-by-time decoding attention visualization. DETAILED DESCRIPTION

[0054] The present invention is described in detail below in conjunction with the accompanying drawings and specific implementation methods. The following description is only for demonstration and explanation and does not limit the present invention in any form.

[0055] The general identification steps have been described in detail in "Summary of the Invention". Now, the identification process of the present invention is specifically described with reference to examples. Figure 1 It is a schematic diagram of the overall processing flow.

[0056] The technical solution of the present invention is mainly divided into the following steps:

[0057] Step 1: Input the text image into the non-autoregressive text recognition NASTR model;

[0058] The non-autoregressive text recognition NASTR model first encodes the text image using an image feature extractor is the corresponding image feature Where H, W are the height and width of the text image, "3" is the number of image channels, d, h, w are the number of channels, height and width of the image features, respectively; furthermore, F I Perform global average pooling calculation to obtain the global features of the image F IPerform a flattening operation to obtain the image visual markup Where m = h × w. The present invention uses Chinese and English STR datasets to evaluate the proposed NASTR. For Chinese, training and evaluation are performed on a large-scale Chinese text recognition dataset. The dataset contains 509,164 scene dataset training samples, 63,645 verification samples, and 63,646 test samples. The Web dataset contains 112,471, 14,059, and 14,059 training, verification, and test samples, respectively. For English text recognition, the present invention is trained on the training set of the recently released benchmark Union14ML, and the performance of NASTR is evaluated on 6 standard benchmarks and the corresponding Union14M-Benchmark test datasets.

[0059] Step 1 specifically includes the following steps:

[0060] 1-1: Using the φ-parameterized Vision Transformer (ViT) model as the image feature extractor, the input scene text image Encoded as the corresponding d-dimensional image feature Where d is set to 384 and patch is set to 4×4. The process can be described as F I =f φ (X).

[0061] 1-2: Perform an average pooling operation along the spatial dimension of the image feature, and pool the feature F with a spatial size of h×w I Transformed into a 1-dimensional vector feature with collapsed spatial information Where d is set to 384 as the global information corresponding to the image. The process can be described as F = AvgPool (F I ).

[0062] 1-3: Flatten the image features along their spatial dimensions to create h×w features F I Transformed into sequence features As input information for the multi-head cross attention layer in the subsequent decoder.

[0063] For Chinese recognition, all text images are resized to 32×256, and no data augmentation is performed for fair comparison with previous methods; for English recognition, all text images are resized to 32×256, and data augmentation effects such as rotation and perspective distortion are performed.

[0064] Step 2: Use the three preset feature encoding constraint components to transform F into different output forms, and make F contain sufficient global image information through additional training objectives. The three preset feature encoding constraint components include a bidirectional image-text global alignment module (ITC) for language knowledge constraints, a length predictor module (LP) for global structure constraints, and a disordered character predictor module (DCP) for global semantic constraints.

[0065] Step 2 is to impose coding constraints on the global feature F of the image during the training phase based on step 1 to solve the problem of quality degradation of F representation caused by language or scene changes. The present invention designs three coding constraint components, and designs corresponding modules from the perspective of language knowledge, global structure, and global semantics, respectively, to improve the coding quality of F through supervised optimization. The specific implementation steps are as follows:

[0066] 2-1: Bidirectional image-text global alignment module based on language knowledge constraints:

[0067] 2-1-1: First, the recognition label Y corresponding to the input text image X is Y = [y1,…,y t ,…,y T ], by embedding the matrix Mapped to the corresponding d-dimensional vector representation Where T and C are the preset maximum character sequence length and the preset number of character categories respectively;

[0068] 2-1-2: Then use the Transformer encoder (including multi-head self-attention layer and forward propagation layer) to transform F T Encoded as contextual semantic features And further Perform average pooling along the time dimension to obtain global text information By aligning F l Combined with the global image information F, the encoded F contains the global language information of the text image, thereby enhancing the quality of subsequent query decoding.

[0069] 2-2: Length predictor module for global structural constraints:

[0070] 2-2-1: First, calculate the number of characters corresponding to the input text image X As the supervisory signal for the length predictor module, L contains the actual number of characters and an additional special marker indicating the end of prediction. For example, if the text content in X is "invention patent", then L = 5.

[0071] 2-2-2: Then use a multilayer perceptron (MLP) with one hidden layer to predict the number of characters in the input text image using the global image information F as input. The predicted value is expressed as a probability vector The process can be described as p l =W (2) σ(W (1) (F)), where Both are learnable linear transformation matrices, and σ(·) is the ReLU activation function.

[0072] 2-3: Unordered character predictor module for global semantic constraints:

[0073] 2-3-1: First, calculate the character frequency distribution vector corresponding to the input text image X relative to the preset character category number C And q c satisfy First, perform deduplication processing on the character content contained in X to obtain the potential character category set S corresponding to X. For example, if the text content in X is "I love invention, invention patent, text recognition, pattern recognition", then S = {I, love, invention, patent, text, character, recognition, pattern}. c It can be calculated by the following formula

[0074]

[0075] Among them, i=1,…,C is the index of each character category, and |S| represents the number of elements in the set.

[0076] 2-3-2: Then use a multilayer perceptron (MLP) with one hidden layer, take the global image information F as input, and predict the existence probability of each character class in the input text image X This process can be described as p c =W (2) σ(W (1) (F)), where Both are learnable linear transformation matrices, and σ(·) is the ReLU activation function.

[0077] Step 3: Copy T copies of F to obtain the image global information query decoding matrix shared by all decoding time steps Where T is the preset maximum decoding time step. q With F I As input, through the decoding module consisting of a multi-head self-attention layer, a multi-head cross-attention layer, and a forward propagation layer, the representation matrix is ​​obtained in Represents the feature representation corresponding to the t-th character.

[0078] Step 3 is based on the global image information F obtained in step 1 and the image visual label F v , construct the global to local character recognition decoding process to achieve text image recognition. The specific steps are as follows:

[0079] 3-1: Copy T copies of the image global feature F obtained in step 1-2 to obtain Let it be the initial input of T decoding time steps, the process can be described as F q =F.unsqueeze(dim=0).repeat(T,1).

[0080] 3-2: Create a position encoding matrix and compare it with F q Superposition. Use sine and cosine fixed position coding to construct the position coding matrix With F q After adding, we get As input to the complete decoder.

[0081] 3-3: Transformation through Multi Head Self-Attention (MHSA) is the time-step specific decoding feature, and the process can be described as Where LN represents the layer normalization operation.

[0082] 3-4: Through the Multi Head Cross-Attention (MHCA) layer, using F d From the image visual mark F v Retrieve character-level feature representations from the dataset and map them into representation matrices through the feed forward network (FFN) This process can be described as F r =FFN(LN(MHCA(F d ,F v )+F d )).

[0083] Step 4: Map F through a linear classifier shared across T time steps r Output for classification Where C represents the number of preset character categories; cls The identification tag corresponding to X Calculating cross entropy loss where y t represents the expected character category at the t-th time step.

[0084] Step 5: By calculating the image global feature encoding constraint loss oriented to the language knowledge constraint Image global feature encoding constraint loss for global structure constraints and image global feature encoding constraint loss for global semantic constraints Further optimize the global image information. Combined with Linear combination and Get the final loss and use back propagation to update the parameters of the non-autoregressive text recognition NASTR model.

[0085] Based on step 2, the losses of the three encoding constraints are calculated to further optimize the global image information F. Combined with the loss results in step 4, the preset hyperparameter λ ITC ,λ LP ,λ DCP Linearly combine the four loss values ​​to get the total loss, and then use the back propagation algorithm to optimize the entire model. The specific implementation steps are as follows:

[0086] 5-1 calculates the losses of the three encoding constraints respectively to further optimize the global image information. The process can be described as:

[0087] 5-1-1: Bidirectional Image-Text Global Alignment Loss

[0088] For a training batch with N text images and N text annotations In the present invention, i is set to 64. Using the process described in step 1 and step 2, the text image X is obtained. i Corresponding global image features and corresponds to the text label Y i Global language features From this we can calculate in, and They are the alignment losses from image to text and text to image, respectively, and are calculated by the following formulas:

[0089]

[0090] in and The global language features and global image features in the batch are enumerated respectively. and They represent the dot product of the ith global image feature and the ith global language feature, respectively, expressing the similarity between the two. The superscripts I and T distinguish the specific directions of the bidirectional alignment calculation. τ is the preset temperature parameter, set to 0.07.

[0091] 5-1-2: Length prediction loss

[0092] As described in step 2-2, length prediction is modeled as a classification problem with T number of categories, so the length prediction loss is Using cross entropy implementation, the probability vector p output by the number of characters L and the length predictor is annotated l For input,

[0093] 5-1-3: Out-of-order character prediction loss

[0094] Take the character frequency distribution q obtained in step 2-3 c and the probability vector p output by the out-of-order character predictor c As input, the unordered character prediction process is modeled as a distribution alignment problem. It is realized through KL divergence (Kullback-LeiblerDivergence). The specific calculation process is:

[0095] Only the average pooling of the features output by the feature extractor is used as the global image information. Without any additional constraints, F faces the problem of being highly similar in different text images (such as scene changes, language changes), which weakens the guidance of the text-specific decoding process. The above three encoding constraint components designed in step 2 can meet the hierarchical requirements of global image features to accommodate information under multi-language and multi-scene changes, and enhance the specific information of the input text image in the global image features. This enhancement improves the distinguishability between global image features from different text images.

[0096] 5-2: By preset hyperparameter λ ITC ,λ LP ,λ DCP Linearly combine the four loss values ​​and calculate the total loss. The process can be described as:

[0097]

[0098] For Scene datasets, set to λ LP =0.1.

[0099] For the Web dataset, only the ITC constraint component is used for training, which is set to λ CE =λITC =1.

[0100] For the English dataset, only the recognition loss is required, which is set to λ CE =1.

[0101] In addition, due to the different dataset sizes, the training epochs for the Scene and Web datasets are set to 100 and 300, respectively.

[0102] 5-3: Using back propagation to update the parameters of the non-autoregressive text recognition NASTR model. The non-autoregressive text recognition NASTR model is composed of the image feature extractor, feature encoding constraint component, decoding module and linear classifier described in steps 1 to 4.

[0103] Step 6: Implement text recognition through the trained non-autoregressive text recognition NASTR model. In the Chinese Scene and Web datasets, NASTR can achieve 78.10% and 69.92% recognition accuracy respectively.

[0104] Ablation study of different combinations of global image feature encoding constraints on Chinese datasets. Here, DCP, LP, and ITC represent the disordered character predictor module for global semantic constraints, the length predictor module for global structural constraints, and the bidirectional image-text global alignment module for language knowledge constraints, respectively.

[0105] DCP LP ITC Scene Web √ √ √ 77.77 69.49 × √ √ 76.20 68.48 × × √ 77.23 69.75 × × × 76.68 68.64

[0106] The impact of different text encoder stacking layers on the accuracy on the Chinese dataset. Here, based on the experimental results of the ablation study of different combinations of global image feature encoding constraints, for the Scene dataset, the LP and DCP components are included in the training, while for the Web dataset, these components are excluded.

[0107]

[0108] The above contents are further detailed descriptions of the present invention in combination with specific / preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, they can also make several substitutions or modifications to these described embodiments without departing from the concept of the present invention, and these substitutions or modifications should be regarded as belonging to the protection scope of the present invention.

[0109] Parts of the present invention that are not described in detail belong to the well-known technologies of those skilled in the art.

Claims

1. A text recognition method based on image global information query decoding, characterized in that: The steps include: Step 1: Encode the input text image using an image feature extractor is the corresponding image feature Where H, W are the height and width of the text image, "3" is the number of image channels, d, h, w are the number of channels, height and width of the image features, respectively; furthermore, F I Perform global average pooling calculation to obtain the global features of the image F I Perform a flattening operation to obtain the image visual markup Where m = h × w; Step 2: transform F into different output forms using three preset feature encoding constraint components, and make F contain sufficient global image information through additional training objectives; the three preset feature encoding constraint components include a bidirectional image-text global alignment module for language knowledge constraints, a length predictor module for global structure constraints, and an unordered character predictor module for global semantic constraints; Step 3: Copy T copies of F to obtain the image global information query decoding matrix shared by all decoding time steps Where T is the preset maximum decoding time step; F q With F I As input, through the decoding module consisting of a multi-head self-attention layer, a multi-head cross-attention layer, and a forward propagation layer, the representation matrix is ​​obtained in Indicates the feature representation corresponding to the t-th character; Step 4: Map F through a linear classifier shared across T time steps r Output for classification Where C represents the number of preset character categories; cls The identification tag corresponding to X Calculating cross entropy loss where y t represents the expected character category at the t-th time step; Step 5: By calculating the image global feature encoding constraint loss oriented to the language knowledge constraint Image global feature encoding constraint loss for global structure constraints and image global feature encoding constraint loss for global semantic constraints Further optimize the global image information; then combine Linear combination and Get the final loss and use back propagation to update the parameters of the non-autoregressive text recognition NASTR model. Step 6: Implement text recognition through the trained non-autoregressive text recognition NASTR model.

2. The text recognition method based on image global information query decoding according to claim 1 is characterized in that: The step 1 specifically includes the following steps: 1-1: Using the φ-parameterized visual Transformer model as the image feature extractor, the input scene text image Encoded as the corresponding d-dimensional image feature This process can be described as F I =f φ (X); 1-2: Perform an average pooling operation along the spatial dimension of the image feature, and pool the feature F with a spatial size of h×w I Transformed into a 1-dimensional vector feature with collapsed spatial information As the global information corresponding to the image, the process can be described as F = AvgPool (FI ) ; 1-3: Flatten the image features along their spatial dimensions to create h×w features F I Transformed into sequence features As input information for the multi-head cross attention layer in the subsequent decoder.

3. The text recognition method based on image global information query decoding according to claim 2 is characterized in that: In step 2, based on step 1, encoding constraints are imposed on the global feature F of the image during the training phase to solve the problem of quality degradation of F representation caused by language or scene changes; three encoding constraint components are designed, and corresponding modules are designed from the perspective of language knowledge, global structure, and global semantics, respectively, to improve the encoding quality of F through supervised optimization. The specific implementation steps are as follows: 2-1: Bidirectional image-text global alignment module based on language knowledge constraints: 2-1-1: First, the recognition label Y = [y1, ..., yt, ..., yT] corresponding to the input text image X is embedded in the matrix Mapped to the corresponding d-dimensional vector representation Where T and C are the preset maximum character sequence length and the preset number of character categories respectively; 2-1-2: Then use the Transformer encoder to transform F T Encoded as contextual semantic features And further Perform average pooling along the time dimension to obtain global text information By aligning F l and the global image information F, so that the encoded F contains the global language information of the text image, thereby enhancing the quality of subsequent query decoding; 2-2: Length predictor module for global structural constraints: 2-2-1: First, calculate the number of characters corresponding to the input text image X As a supervision signal for the length predictor module, where L contains the actual number of characters and an additional special token indicating the end of the prediction; 2-2-2: Then use a multilayer perceptron with one hidden layer to predict the number of characters in the input text image using the global image information F as input; the predicted value is expressed as a probability vector The process can be described as p l =W (2) σ(W (1) (F)), where Both are learnable linear transformation matrices, and σ(·) is the ReLU activation function; 2-3: Unordered character predictor module for global semantic constraints: 2-3-1: First, calculate the character frequency distribution vector corresponding to the input text image X relative to the preset character category number C And q c satisfy First, the characters contained in X are deduplicated to obtain the potential character category set S corresponding to X. c It can be calculated by the following formula: Where i=1,..., C is the index of each character category, |S| represents the number of elements in the set; 2-3-2: Then use a multi-layer perceptron MLP with one hidden layer, take the global image information F as input, and predict the existence probability of each character class in the input text image X This process can be described as p c =W (2) σ(W (1) (F)), where Both are learnable linear transformation matrices, and σ(·) is the ReLU activation function.

4. A text recognition method based on image global information query decoding according to claim 2 or 3, characterized in that: The specific steps of step 3 are as follows: 3-1: Copy T copies of the image global feature F obtained in step 1-2 to obtain Let it be the initial input of T decoding time steps, the process can be described as F q =F.unsqueeze(dim=0).repeat(T, 1); 3-2: Create a position encoding matrix and compare it with F q Superposition; Use sine and cosine fixed position coding to construct the position coding matrix With F q After adding, we get As a complete decoder input; 3-3: Transformation through Multi-Head Self-Attention Layer MHSA is the time-step specific decoding feature, and the process can be described as Where LN represents the layer normalization operation; 3-4: Through the multi-head cross attention layer MHCA, using F d From the image visual mark F v Retrieve character-level feature representations from the CNN and map them into representation matrices through the forward propagation layer FFN This process can be described as F r =FFN(LN(MHCA(F d , F v )+F d )).

5. The text recognition method based on image global information query decoding according to claim 4 is characterized in that: The specific implementation steps of step 5 are as follows: 5-1 calculates the losses of the three encoding constraints respectively to further optimize the global image information. The process can be described as: 5-1-1: Bidirectional Image-Text Global Alignment Loss For a training batch with N text images and N text annotations Using the process described in steps 1 and 2, we get the text image X i Corresponding global image features and corresponds to the text label Y i Global language features From this we can calculate in, and They are the alignment losses from image to text and text to image, respectively, and are calculated by the following formulas: in and The global language features and global image features in the batch are enumerated respectively. and They represent the dot product of the ith global image feature and the ith global language feature, respectively, expressing the similarity between the two. The superscripts I and T distinguish the specific directions of the bidirectional alignment calculation; τ is the preset temperature parameter; 5-1-2: Length prediction loss Length prediction loss It is implemented using cross entropy, with the character number label L and the probability vector pl output by the length predictor as input. 5-1-3: Out-of-order character prediction loss Character frequency distribution q c and the probability vector p output by the out-of-order character predictor c As input, the unordered character prediction process is modeled as a distribution alignment problem. It is achieved through KL divergence. The specific calculation process is: 5-2: By preset hyperparameter λ ITC ,λ LP ,λ DCP Linearly combine the four loss values ​​and calculate the total loss. The process can be described as: 5-3: Use back propagation to update the parameters of the non-autoregressive text recognition NASTR model; The non-autoregressive text recognition NASTR model is composed of the image feature extractor, feature encoding constraint component, decoding module and linear classifier described in steps 1 to 4.

Images