Text recognition method and device and storage medium

Abstract

The invention discloses a text recognition method and device and a storage medium, and the method comprises the steps: carrying out the image feature extraction of a to-be-recognized image, and obtaining an image feature vector; calculating the probability that each feature in the image feature vector belongs to the text region, and determining a text attention region based on the probability that each feature in the image feature vector belongs to the text region; performing position coding on the text attention area to obtain a query vector, and extracting an image feature corresponding to the text attention area to obtain a window feature vector; and decoding the text category and the text position based on the query vector and the window feature vector to obtain a text recognition result of the to-be-recognized text in the to-be-recognized image. The position information of the region needing to be concerned and the image information of the region needing to be concerned are integrated together and transmitted to the decoder, interaction between the query vector and irrelevant background information can be reduced, interference of other irrelevant region features in the image on text decoding is reduced, and the accuracy of a finally output text recognition result is improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a text recognition method, device and storage medium. Background Technology

[0002] Scene text recognition is a technique that locates and identifies text appearing in a scene image, facilitating further analysis of the image content.

[0003] Scene text recognition mainly involves two tasks: locating and recognizing text in an image. Currently, commonly used recognition techniques are based on deep learning methods. Generally, a detection algorithm is first used to output bounding boxes for all text in the image. Each text box region is then cropped, corrected, and enhanced before a character recognition algorithm is used to identify each character in the text. However, the above method may introduce additional background noise during the cropping stage, affecting the accuracy of subsequent character recognition. Summary of the Invention

[0004] To address the aforementioned technical problems, this application provides at least one text recognition method, device, and storage medium.

[0005] The first aspect of this application provides a text recognition method, which includes: acquiring an image to be recognized; extracting image features from the image to be recognized to obtain an image feature vector; wherein the image to be recognized contains text to be recognized; calculating the probability that each feature in the image feature vector belongs to a text region; determining a text interest region based on the probability that each feature in the image feature vector belongs to a text region; performing position encoding on the text interest region to obtain a query vector; extracting image features corresponding to the text interest region to obtain a window feature vector; and decoding the text category and text position based on the query vector and the window feature vector to obtain the text recognition result of the text to be recognized in the image to be recognized.

[0006] In one embodiment, calculating the probability that each feature in the image feature vector belongs to a text region, and determining the text interest region based on the probability that each feature in the image feature vector belongs to a text region, includes: calculating the probability that each feature in the image feature vector belongs to a text region, selecting features in the image feature vector whose probabilities satisfy preset conditions to obtain key features; calculating the coordinate offset and window width and height of the position of each text to be identified relative to each feature in the image feature vector, and calculating the window information corresponding to each text to be identified relative to each feature in the image feature vector based on the position, coordinate offset, and window width and height of the features; and using the window information corresponding to each key feature in the image feature vector as the text interest region.

[0007] In one embodiment, calculating the probability that each feature in the image feature vector belongs to a text region includes: inputting the image feature vector into a classification linear layer to obtain the probability that each feature in the image feature vector output by the classification linear layer belongs to a text region; calculating the coordinate offset and window width and height of each text to be identified relative to each feature in the image feature vector includes: inputting the image feature vector into a local window converter composed of a multilayer perceptron to obtain the coordinate offset and window width and height of each text to be identified relative to each feature in the image feature vector output by the local window converter.

[0008] In one embodiment, the method of obtaining a query vector by position encoding the text region of interest includes: obtaining a position query vector by position encoding the text region of interest; obtaining a content query vector by text encoding a preset sequence start character; and obtaining the final query vector by combining the position query vector and the content query vector.

[0009] In one embodiment, extracting image features corresponding to the text interest region to obtain a window feature vector includes: extracting image features corresponding to the text interest region to obtain initial window features; and interpolating the initial window features corresponding to each text interest region according to a uniform width and height to obtain a window feature vector corresponding to each text interest region.

[0010] In one embodiment, a text recognition model is pre-trained. The text recognition model includes an encoder and a decoder. The encoder encodes the image to be recognized to obtain a query vector and a window feature vector. The decoder decodes the query vector and the window feature vector to obtain the text recognition result of the text to be recognized in the image.

[0011] In one embodiment, the encoder includes a feature extraction layer, a query vector construction layer, and a local window feature construction layer. The image to be recognized is input to the feature extraction layer for image feature extraction to obtain an image feature vector. The image feature vector is input to the query vector construction layer to calculate the probability that each feature belongs to a text region, thereby obtaining the text attention region and query vector output by the query vector construction layer. The image feature vector and the text attention region are input to the local window feature construction layer to obtain the window feature vector output by the local window feature construction layer.

[0012] In one embodiment, the decoder includes a self-attention layer, a cross-attention layer, a feedforward neural network layer, and a classification layer. Based on a query vector and a window feature vector, it decodes the text category and text location to obtain the text recognition result of the text to be recognized in the image to be recognized. This includes: encoding the query vector into text location and inputting it into the self-attention layer to obtain a self-attention query vector output by the self-attention layer; performing a linear transformation on the window feature vector to obtain a value vector and a key vector; inputting the value vector, key vector, and self-attention query vector into the cross-attention layer to obtain a feature vector output by the cross-attention layer; inputting the feature vector output by the cross-attention layer into the feedforward neural network layer for a non-linear transformation to obtain a feature vector output by the feedforward neural network layer; and inputting the feature vector output by the feedforward neural network layer into the classification layer to obtain the text category and text location of the text to be recognized output by the classification layer, thus obtaining the text recognition result of the text to be recognized in the image to be recognized.

[0013] In one embodiment, the training steps of the text recognition model include: acquiring image samples labeled with text type labels and text location labels; inputting the image samples into an encoder to be trained to obtain the probability prediction results of each feature output by the encoder to be trained belonging to a text region, the text interest region prediction results, the query vector, and the window feature vector; inputting the query vector and the window feature vector into a decoder to be trained to obtain the text category prediction results output by the decoder to be trained; calculating the probability prediction loss based on the probability prediction results and the text location labels, and calculating the text region prediction loss based on the text interest region prediction results and the text location labels; calculating the category prediction loss based on the text category prediction results and the text type labels; updating the encoder to be trained using the probability prediction loss and the text region prediction loss to obtain the trained encoder; and updating the decoder to be trained using the category prediction loss to obtain the trained decoder.

[0014] A second aspect of this application provides a text recognition apparatus, comprising: a feature extraction module for acquiring an image to be recognized and extracting image features from the image to obtain an image feature vector; wherein the image to be recognized contains text to be recognized; a region determination module for calculating the probability that each feature in the image feature vector belongs to a text region and determining a text interest region based on the probability that each feature in the image feature vector belongs to a text region; a position and local feature encoding module for encoding the text interest region at a position to obtain a query vector and extracting image features corresponding to the text interest region to obtain a window feature vector; and a decoding module for decoding the text category and text position based on the query vector and the window feature vector to obtain the text recognition result of the text to be recognized in the image to be recognized.

[0015] A third aspect of this application provides an electronic device, including a memory and a processor, wherein the processor is configured to execute program instructions stored in the memory to implement the text recognition method described above.

[0016] The fourth aspect of this application provides a computer-readable storage medium having program instructions stored thereon, which, when executed by a processor, implement the above-described text recognition method.

[0017] The above scheme extracts image features from the image to be recognized to obtain an image feature vector; calculates the probability that each feature in the image feature vector belongs to a text region, and determines the text interest region based on the probability of each feature belonging to a text region; performs positional encoding on the text interest region to obtain a query vector, and extracts the image features corresponding to the text interest region to obtain a window feature vector; decodes the text category and text position based on the query vector and window feature vector to obtain the text recognition result of the text to be recognized in the image to be recognized. Integrating the positional information of the region of interest with the image information of the region of interest together and inputting it into the decoder can reduce the interaction between the query vector and irrelevant background information, reduce the interference of other irrelevant region features in the image on text decoding, and improve the accuracy of the final output text recognition result.

[0018] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0020] Figure 1 This is a schematic diagram illustrating the implementation environment of the solution in an exemplary embodiment of this application; Figure 2 This is a flowchart illustrating a text recognition method in an exemplary embodiment of this application; Figure 3 This is a schematic diagram illustrating a text recognition model as shown in an exemplary embodiment of this application; Figure 4 This is a schematic diagram illustrating how a query vector is obtained by encoding a query vector construction layer, as shown in an exemplary embodiment of this application. Figure 5 This is a schematic diagram illustrating a local window feature construction layer that obtains a window feature vector, as shown in an exemplary embodiment of this application. Figure 6 This is a schematic diagram illustrating a decoder performing decoding, as shown in an exemplary embodiment of this application; Figure 7This is a block diagram illustrating a text recognition device in an exemplary embodiment of this application; Figure 8 This is a schematic diagram of the structure of an electronic device shown in an exemplary embodiment of this application; Figure 9 This is a schematic diagram illustrating the structure of a computer-readable storage medium, as shown in an exemplary embodiment of this application. Detailed Implementation

[0021] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0022] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0023] In this document, the term "and / or" is merely a description of the association information of related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this document means two or more. Moreover, the term "at least one" in this document means any combination of at least two of any one or more of a plurality of elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0024] The text recognition method provided in the embodiments of this application will be described below.

[0025] Please refer to Figure 1 , Figure 1 This is a schematic diagram illustrating an implementation environment of the scheme according to an exemplary embodiment of this application. The implementation environment may include a terminal 110 and a server 120, and the terminal 110 and the server 120 are interconnected.

[0026] The number of terminals 110 can be one or more. Terminals 110 can be cameras, smartphones, tablets, laptops, desktop computers, smartwatches, etc., but are not limited to these.

[0027] Server 120 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0028] In one example, server 120 can perform text recognition processing on the image to be recognized obtained from terminal 110 to obtain the text recognition result. Of course, server 120 can store the text recognition result locally, send it back to terminal 110, or transmit it to other terminals.

[0029] In one example, a client application with a target application is installed and running on terminal 110. This target application may be an application that provides text recognition functionality. Terminal 110 performs text recognition processing on the image to be recognized by running the target application and obtains the text recognition result. Server 120 may be a backend server for the target application, used to provide background services to the client of the target application.

[0030] The text recognition method provided in this application embodiment can be executed by a terminal 110, such as the client of the target application installed and running on the terminal 110, or by a server 120, or by the interaction and cooperation between the terminal 110 and the server 120, that is, some steps of the method are executed by the terminal 110 and other steps are executed by the server 120.

[0031] Please see Figure 2 , Figure 2 This is a flowchart illustrating a text recognition method in an exemplary embodiment of this application. This text recognition method can be applied to… Figure 1 The implementation environment shown is specifically executed by a server within that implementation environment. It should be understood that this method can also be applied to other exemplary implementation environments and executed by devices in other implementation environments; this embodiment does not limit the implementation environment to which the method is applicable.

[0032] like Figure 2 As shown, the text recognition method includes at least steps S210 to S240, which are described in detail below: Step S210: Obtain the image to be recognized, extract image features from the image to be recognized, and obtain the image feature vector; wherein, the image to be recognized contains text to be recognized.

[0033] The image to be recognized is the image that needs to be recognized for text recognition. It can be an RGB image, a YUV image, an infrared image, or a grayscale image, etc. This application does not limit the image type of the image to be recognized.

[0034] The image to be recognized contains text to be recognized. The goal of text recognition is to identify the text type and / or text location of the text to be recognized in the image.

[0035] Image features are extracted from the image to be recognized to obtain the image feature vector.

[0036] Image feature extraction is used to extract information from the image to be identified that can effectively characterize the content or structure of the image, such as extracting at least one of the following: color features, texture features, shape features, spatial features, and abstract features generated through mathematical statistics or transformation.

[0037] For example, image features can be extracted from the image to be recognized based on traditional mathematical methods, such as using the Canny operator, and / or the Sobel operator, and / or the Harris algorithm, to obtain the image feature vector.

[0038] For example, deep learning methods can be used to extract image features from the image to be recognized, such as using neural networks that can be used for image feature extraction, such as Convolutional Neural Networks (CNN) and / or Residual Networks (ResNet), to extract image features from the image to be recognized and obtain image feature vectors.

[0039] For example, traditional mathematical methods and deep learning methods can be combined to extract image features from the image to be recognized, thereby obtaining an image feature vector.

[0040] It should be noted that the specific implementation method of image feature extraction can be flexibly selected according to the actual application scenario, and this application does not limit it.

[0041] Step S220: Calculate the probability that each feature in the image feature vector belongs to the text region, and determine the text interest region based on the probability that each feature in the image feature vector belongs to the text region.

[0042] Deep learning is used to select a pre-trained neural network model. The neural network model directly learns the mapping from features to probabilities end-to-end. Using the pre-trained neural network model, the probability that each feature in the image feature vector belongs to the text region is calculated.

[0043] Specifically, an image feature vector consists of multiple features, and a single feature can correspond to one or more pixels in the image to be identified. A pre-trained neural network model is used to identify the probability that each feature in the image feature vector belongs to a text region.

[0044] The higher the probability that a feature belongs to a text region, the higher the probability that the pixel corresponding to that feature is located within the text region; conversely, the lower the probability that a feature belongs to a text region, the lower the probability that the pixel corresponding to that feature is located within the text region. Therefore, the text interest region is determined based on the probability that each feature in the image feature vector belongs to a text region.

[0045] For example, the image region corresponding to the feature with a probability greater than a preset probability threshold can be taken as the text attention region, or the image region corresponding to the top k features with the highest probability (k≥1) can be taken as the text attention region. This application does not limit this.

[0046] Step S230: Encode the text region of interest to obtain the query vector, and extract the image features corresponding to the text region of interest to obtain the window feature vector.

[0047] The text interest region is positionally encoded to vectorize the positional information of the text interest region.

[0048] Cosine position coding or sine position coding can be used to position the regions of interest in the text. This application does not limit the specific implementation of position coding.

[0049] In addition, the image features corresponding to the text interest region are extracted to obtain the window feature vector corresponding to the text interest region.

[0050] It should be noted that if there are multiple regions of interest in the text, image features can be extracted for each region of interest separately to obtain multiple window feature vectors.

[0051] Step S240: Decode the text category and text location based on the query vector and window feature vector to obtain the text recognition result of the text to be recognized in the image to be recognized.

[0052] The query vector indicates the location of the region of interest in the image to be recognized, and the window feature vector contains image information at the location of the region of interest. By combining the query vector and the window feature vector for decoding, a more accurate text category and text location are obtained, resulting in the text recognition result of the text to be recognized in the image.

[0053] This application extracts window feature vectors and combines them with query vectors and window feature vectors for text decoding. By integrating the location information of the region of interest with the image information of the region of interest into the decoder, the interaction between the query vector and irrelevant background information can be reduced, the interference of other irrelevant region features in the image on text decoding can be reduced, and the accuracy of the final output text recognition result can be improved.

[0054] The following describes some embodiments of this application in detail.

[0055] In some implementations, step S220 calculates the probability that each feature in the image feature vector belongs to the text region, and determines the text interest region based on the probability that each feature in the image feature vector belongs to the text region, including the following steps S221 to S223.

[0056] Step S221: Calculate the probability that each feature in the image feature vector belongs to the text region, select the features in the image feature vector whose probability meets the preset conditions, and obtain the key features.

[0057] For example, by inputting the image feature vector into the classification linear layer, we can obtain the probability that each feature in the image feature vector output by the classification linear layer belongs to the text region.

[0058] Among them, the linear layer is used to perform linear transformation on the input data. In this application, the classification linear layer is used to perform linear transformation on each feature in the input image feature vector to map the probability that each feature belongs to the text region.

[0059] Key features are obtained by selecting features from the image feature vectors whose probabilities meet preset conditions.

[0060] For example, features with a probability greater than a preset probability threshold can be used as features that meet preset conditions to obtain key features. Alternatively, the top k features with the highest probability (k≥1) can be used as features that meet preset conditions to obtain key features.

[0061] Step S222: Calculate the coordinate offset and window width and height of each text to be recognized relative to each feature in the image feature vector. Based on the position, coordinate offset and window width and height of the feature, calculate the window information corresponding to each feature in the image feature vector for each text to be recognized.

[0062] Calculate the coordinate offset of each text to be recognized relative to the position of each feature in the image feature vector and the width and height of the window.

[0063] Among them, window width and height are used to describe the size of the text area, coordinate offset is used to describe the offset of the text area coordinates relative to the position of the feature, and text area coordinates can refer to the center coordinates of the text area.

[0064] For example, the image feature vector is input into a local window converter composed of a multilayer perceptron, and the coordinate offset and window width and height of each text to be recognized relative to the position of each feature in the image feature vector are obtained from the output of the local window converter.

[0065] The local window transformer can divide the input image feature vector into multiple local windows and compute the feature representation independently within each window.

[0066] Multilayer perceptrons (MLPs) can learn complex input-output mapping relationships through nonlinear activation functions. MLPs can be used to generate position bias values ​​to obtain coordinate biases.

[0067] Based on the feature's location, coordinate offset, and window width and height, the window information corresponding to each feature in the image feature vector for each text to be recognized is calculated.

[0068] Specifically, the position of the feature and the coordinate offset calculated for the feature are added together to obtain the window coordinates corresponding to the feature. The window information is obtained by combining the window coordinates and the window width and height.

[0069] Step S223: Use the window information corresponding to each key feature in the image feature vector as the text attention region.

[0070] Obtain the window information calculated for each key feature, use it as the text attention region, and then perform position encoding on the text attention region to obtain the query vector.

[0071] In some implementations, step S230 involves position encoding the text region of interest to obtain a query vector, including steps S2311 to S2313 below.

[0072] Step S2311: Perform position encoding on the text interest region to obtain the position query vector.

[0073] By encoding the regions of interest in the text, a location query vector is obtained, which provides text location information during subsequent decoding.

[0074] Step S2312: Text-encode the preset sequence start character to obtain the content query vector.

[0075] For example, the preset sequence start character can be <sos>Of course, the sequence start symbol can also be represented in other ways, and this application does not limit this.

[0076] Text encoding is performed on the preset sequence start character to obtain the content query vector.

[0077] Step S2313: Combine the location query vector and the content query vector to obtain the final query vector.

[0078] By fusing the sequence start symbol and text position information of the text into the query vector, the sequence start symbol indicates the start of sequence prediction and serves as a trigger signal for the decoder to start generating the sequence. This can effectively guide the prediction of text position and text content during the decoding process.

[0079] In some implementations, step S230 extracts the image features corresponding to the text interest region to obtain the window feature vector, including the following steps S2321 to S2322.

[0080] Step S2321: Extract the image features corresponding to the text interest region to obtain the initial window features.

[0081] Step S2322: According to the uniform width and height, interpolate the initial window features corresponding to each text interest region to obtain the window feature vector corresponding to each text interest region.

[0082] The uniform width and height dimensions can be preset based on experience or calculated flexibly. For example, the width and height of each text attention area can be statistically analyzed to obtain the maximum width and maximum height of each text attention area, and the maximum width and maximum height can be used as the uniform width and height dimensions.

[0083] Using uniform width and height dimensions, interpolation is performed on the initial window features corresponding to each text interest region to ensure that the width and height dimensions of the interpolated window feature vectors are the same.

[0084] The interpolation methods include, but are not limited to, bilinear interpolation, linear interpolation, and nearest neighbor interpolation.

[0085] By using interpolation, the regional features of each text are unified to the same scale, enabling the model to give equal attention to text instances of different sizes, which can effectively improve the recognition effect of small-scale text.

[0086] Then, the query vector and window feature vector are input into the decoder to decode the text category and text location, and obtain the text recognition result of the text to be recognized in the image to be recognized.

[0087] In some implementations, a text recognition model is pre-trained. The text recognition model includes an encoder and a decoder. The encoder encodes the image to be recognized to obtain a query vector and a window feature vector. The decoder decodes the query vector and the window feature vector to obtain the text recognition result of the image to be recognized.

[0088] For example, please refer to Figure 3 , Figure 3 This is a schematic diagram illustrating a text recognition model as shown in an exemplary embodiment of this application, such as... Figure 3 As shown, the encoder includes a feature extraction layer, a query vector construction layer, and a local window feature construction layer. The image to be recognized is input into the encoder for encoding processing to obtain the query vector and the window feature vector. The query vector and the window feature vector are then input into the decoder for decoding processing to obtain the text recognition result.

[0089] For example, the feature extraction layer includes a convolutional neural network (CNN) and a Transformer encoder. The image to be recognized is input into the feature extraction layer to extract image features and obtain an image feature vector.

[0090] Specifically, a CNN backbone network is used to extract multi-scale features from the image to be recognized. Then, the number of channels in the multi-scale feature maps is unified through convolution operations. Finally, the multi-scale features are flattened and stitched together to obtain the initial image features. Where B represents the batch size, S represents the sum of the width and height products of multiple scale feature maps, and C represents the number of feature channels. The CNN backbone network can be any network structure capable of image feature extraction, such as ResNet, SwinTransformer (Shifted Window Transformer), ViT (Vision Transformer), etc.

[0091] The initial image features F are then fed into the Transformer encoder. The Transformer encoder learns the global information of the image, adjusts the focus of features at different locations in the initial image features F, strengthens key features, and weakens irrelevant features to obtain the final image feature vector. .

[0092] Then, the image feature vector The input is fed into the query vector construction layer to calculate the probability that each feature belongs to the text region, and the text attention region and query vector output by the query vector construction layer are obtained.

[0093] For example, see Figure 4 , Figure 4 This is a schematic diagram illustrating how a query vector is obtained by encoding a query vector in a query vector construction layer, as shown in an exemplary embodiment of this application. Figure 4 As shown, image feature vector The input is fed into a classification linear layer, which outputs an image feature vector. The probability of each feature belonging to a text region. This yields the indices of the top k features ranked from highest to lowest probability. Also, the image feature vectors... The input is fed into a local window transformer composed of multiple sensors. The local window transformer outputs the coordinate offset of the position of each text to be recognized relative to each feature. Given the window width and height (w, h), we can obtain... By adding the corresponding coordinate offset to the position of each feature, the center coordinates of the text region predicted by that feature can be obtained, thus yielding the corresponding window information. Based on the indices of the top k features ranked from highest to lowest probability, the window information corresponding to the top k features is obtained. .Will After cosine position encoding, the input is processed by a position encoder composed of an MLP to obtain the position query vector. .

[0094] And, for the preset sequence start character <sos>The corresponding character vector is repeated k times to obtain the content query vector. .

[0095] Location query vector and content query vector Perform the addition calculation to obtain the final query vector. .

[0096] In addition, the image feature vector and text focus area The input is fed into the local window feature construction layer to obtain the window feature vector output by the local window feature construction layer.

[0097] For example, see Figure 5 , Figure 5 This is a schematic diagram illustrating a local window feature construction layer that obtains a window feature vector, as shown in an exemplary embodiment of this application. Figure 5 As shown, based on the window information corresponding to the features of the first k... This yields the shape of the local window corresponding to the k query vectors, derived from the image feature vectors. The initial window features were obtained from the interception. Interpolation is used to transform the features of each initial window. Interpolation is Let g be the width and height of the interpolated window. To unify the width and height dimensions of all window features, we obtain the window feature vector. Then, we flatten each window feature vector to obtain a feature sequence of length g×g. .

[0098] The query vector of the i-th text attention region and window feature vector The input is fed to the decoder for decoding, where 1≤i≤k.

[0099] For example, the decoder includes a self-attention layer, a cross-attention layer, a feedforward neural network layer, and a classification layer; wherein, the self-attention layer, the cross-attention layer, and the feedforward neural network layer constitute the decoding layer, and the decoder may contain N decoding layers.

[0100] The query vector is encoded by text position and then input into the self-attention layer to obtain the self-attention query vector output by the self-attention layer. The window feature vector is linearly transformed to obtain the value vector and key vector. The value vector, key vector, and self-attention query vector are input into the cross-attention layer to obtain the feature vector output by the cross-attention layer. The feature vector output by the cross-attention layer is input into the feedforward neural network layer for nonlinear transformation to obtain the feature vector output by the feedforward neural network layer. The feature vector output by the feedforward neural network layer is input into the classification layer to obtain the text category and text position of the text to be recognized, thus obtaining the text recognition result of the image to be recognized.

[0101] For example, see Figure 6 , Figure 6 This is a schematic diagram illustrating the decoder performing decoding, as shown in an exemplary embodiment of this application. Figure 6 As shown, the query vector for the i-th text attention region The text positional encoding is then input into the decoder. The self-attention layer is specifically causal self-attention, and the query vector after text positional encoding... After passing through the causal self-attention module and Add&Norm (residual connections and layer normalization), it is used as the text query vector. In the input cross-attention module, and the window feature vector The value vector obtained after the linear layer and key vector Cross-attention is computed to reduce interference from irrelevant regions. Specifically, a causal self-attention mask is used to block inputs after the current position in the sequence, restricting the model to accessing only sequence features from the current position and previous positions when calculating the self-attention score.

[0102] The output of the cross-attention module is then passed through a feedforward neural network (FFN) and Add&Norm before being output to the decoding layer, resulting in... , Input the next decoding layer, where L is the text length. Output the last decoding layer. The input is fed into a classification layer, which consists of a linear layer and a softmax (activation function) layer. The query vector is then output by the softmax layer. Text category confidence score after decoding .

[0103] in, To determine the size of the feature dictionary, for example, the text's position coordinates are discretized using categories, i.e., position coordinate values ​​belong to "1" to "2". One of the categories in "", This represents the number of units after discretizing the image's width and height. The total number of character categories in the text content is... Add the "unknown" class, background class, and sequence start character. <sos>Deadline <eos>Therefore, the feature dictionary size used in this application is... The dictionary vectors are updated along with the training of the text recognition network.

[0104] Post-processing is performed on the output of the last layer of the decoder to obtain the text recognition result of the image to be recognized. The specific post-processing is as follows: The last dimension of the text category confidence score C represents the confidence score of the text category at each position in the sequence. The index corresponding to the maximum confidence score is the text category index at that position. The first two characters predicted by each query vector represent the coordinates (x, y) of the point at the position of the text to be recognized, and the subsequent characters represent the text category. The decoded text... <eos>This means that all text predictions have been completed. It should be noted that the output characters x and y are in... The coordinates under the image size are based on the size of the original image to be recognized and... The image scale can be used to transform the coordinates back to the original image to be recognized.

[0105] In some implementations, the training steps of the text recognition model include: acquiring image samples labeled with text type labels and text location labels; inputting the image samples into the encoder to be trained to obtain the probability prediction results of each feature belonging to the text region, the text interest region prediction results, the query vector, and the window feature vector output by the encoder to be trained; inputting the query vector and the window feature vector into the decoder to be trained to obtain the text category prediction results output by the decoder to be trained; calculating the probability prediction loss based on the probability prediction results and the text location labels, and calculating the text region prediction loss based on the text interest region prediction results and the text location labels; calculating the category prediction loss based on the text category prediction results and the text type labels; updating the encoder to be trained using the probability prediction loss and the text region prediction loss to obtain the trained encoder; and updating the decoder to be trained using the category prediction loss to obtain the trained decoder.

[0106] Obtain scene text image data containing text location and text type annotations to obtain image samples. Image samples are image data containing text; for example, text location labels include the center point coordinates and the width and height of the smallest bounding rectangle for each text in the image sample, and text type labels include the text content for each text in the image sample. For ease of training, a padding character equal to the cutoff is used. <pad>Fill text type labels to a uniform length L.

[0107] Image samples are input into the encoder to be trained. The encoder outputs the probability prediction results of each feature belonging to the text region, the text attention region prediction results, the query vector, and the window feature vector. The encoder inference process is similar to the inference process corresponding to the above embodiments.

[0108] During training, after obtaining the probability prediction results of the text region, the text attention region prediction results, the query vector, and the window feature vector, the probability prediction results of k features, the text attention region prediction results, the text type label, and the text position label are used to perform text type label matching for each query vector. Query vectors that match are assigned the corresponding text type label, while query vectors that do not match are assigned a cutoff label. <eos>The matched text, serving as background text, contains L characters. The matching method can include algorithms such as the Hungarian algorithm.

[0109] Then, the L matched characters are mapped to obtain their corresponding feature representations using the feature dictionary, and the query vector corresponding to the image sample is then used. The feature representations of L characters are concatenated to obtain .right After text position encoding, the data is input to the decoder. In other words, the input during training is a query vector concatenated with the feature representations of the matched characters.

[0110] After text position encoding After passing through the causal self-attention module and Add&Norm, it is used as the text query vector. In the input cross-attention module, and the window feature vector The value vector obtained after the linear layer and key vector By calculating cross-attention and learning visual features related to the text instance, the learning scope of the query vector is effectively narrowed to the local area of ​​the text, reducing the interference of irrelevant areas on the overall learning process.

[0111] Similar to the inference process, the output of the cross-attention module is then passed through a feedforward neural network and Add&Norm before being output to the decoding layer. The output of the final decoding layer is then fed into a linear layer and a softmax layer to obtain the query vector output by the softmax layer. Text category confidence score after decoding This yields the text category prediction results.

[0112] During training, the k sets of output sequences from the decoder are labeled with the text type label sequences of all position coordinates of the corresponding image samples. Matching sequences are assigned to the corresponding text type label sequences, while unmatched sequences are assigned to the cutoff sequence. <eos>As background text. During inference, the decoder predicts the next text category by iteratively inputting the next query vector and the text categories decoded in previous loops, until a cutoff is predicted. <eos>So far, this represents the text recognition result for the image to be recognized.

[0113] A loss function is constructed, specifically including: for the probability prediction result, the difference between the probability prediction result and the text position label is calculated using a classification loss function to obtain the probability prediction loss. The classification loss function includes, but is not limited to, Focal Loss classification loss function, cross-entropy loss function, Hinge Loss classification loss function, etc., and this application does not limit it.

[0114] For the text attention region prediction results, the difference between the text attention region prediction results and the text location labels is calculated using the L1 norm loss function and / or the Generalized Intersection over Union Loss (GIoU Loss) function, and the text region prediction loss is obtained.

[0115] For the text category prediction results, the difference between the text category prediction results and the text type labels is calculated using the maximum likelihood loss function and / or mean squared error loss function to obtain the text region prediction loss.

[0116] The encoder to be trained is updated based on probability prediction loss and text region prediction loss to obtain the trained encoder. The decoder to be trained is updated based on category prediction loss to obtain the trained decoder. Of course, the encoder to be trained can also be updated based on category prediction loss. This application does not limit this. The gradient information of the decoder decoding process is passed back to the query vector construction module to help adjust the window information of the text, so that the network can adaptively learn how to more accurately locate the text region and pass the features with higher relevance to the text to the text decoder.

[0117] Figure 7 This is a block diagram illustrating a text recognition device in an exemplary embodiment of this application. Figure 7 As shown, the exemplary text recognition device 700 includes: The feature extraction module 710 is used to acquire the image to be recognized, extract image features from the image to be recognized, and obtain an image feature vector; wherein, the image to be recognized contains text to be recognized; The region determination module 720 is used to calculate the probability that each feature in the image feature vector belongs to the text region, and to determine the text interest region based on the probability that each feature in the image feature vector belongs to the text region. The location and local feature encoding module 730 is used to encode the text interest region to obtain the query vector and extract the image features corresponding to the text interest region to obtain the window feature vector. The decoding module 740 is used to decode the text category and text location based on the query vector and window feature vector to obtain the text recognition result of the text to be recognized in the image to be recognized.

[0118] It should be noted that the text recognition device and the text recognition method provided in the above embodiments belong to the same concept. The specific ways in which each module and unit performs operations have been described in detail in the method embodiments, and will not be repeated here. In practical applications, the text recognition device provided in the above embodiments can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. This is not a limitation.

[0119] Please see Figure 8 , Figure 8 This is a schematic diagram illustrating the structure of an electronic device in an exemplary embodiment of this application. The electronic device 800 includes a memory 801 and a processor 802. The processor 802 executes program instructions stored in the memory 801 to implement the steps in any of the above-described text recognition method embodiments. In a specific implementation scenario, the electronic device 800 may include, but is not limited to, a microcomputer or a server. Furthermore, the electronic device 800 may also include mobile devices such as laptops and tablets, without limitation.

[0120] Specifically, processor 802 controls itself and memory 801 to implement the steps in any of the above-described text recognition method embodiments. Processor 802 can also be referred to as a Central Processing Unit (CPU). Processor 802 may be an integrated circuit chip with signal processing capabilities. Processor 802 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 802 can be implemented using integrated circuit chips.

[0121] Please see Figure 9 , Figure 9 This is a schematic diagram illustrating the structure of a computer-readable storage medium in an exemplary embodiment of this application. The computer-readable storage medium 900 stores program instructions 910 that can be executed by a processor, the program instructions 910 being used to implement the steps in any of the above-described text recognition method embodiments.

[0122] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0123] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0124] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0125] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.< / eos> < / eos> < / eos> < / pad> < / eos> < / eos> < / sos> < / sos> < / sos>

Claims

1. A text recognition method, characterized in that, The method includes: An image to be identified is acquired, and image features are extracted from the image to be identified to obtain an image feature vector; wherein, the image to be identified contains text to be identified; Calculate the probability that each feature in the image feature vector belongs to the text region, and determine the text attention region based on the probability that each feature in the image feature vector belongs to the text region; The text interest region is positionally encoded to obtain a query vector, and the image features corresponding to the text interest region are extracted to obtain a window feature vector. Based on the query vector and the window feature vector, the text category and text position are decoded to obtain the text recognition result of the text to be recognized in the image to be recognized.

2. The method according to claim 1, characterized in that, The step of calculating the probability that each feature in the image feature vector belongs to a text region, and determining the text interest region based on the probability that each feature in the image feature vector belongs to a text region, includes: Calculate the probability that each feature in the image feature vector belongs to the text region, and select the features in the image feature vector whose probabilities satisfy preset conditions to obtain the key features; Calculate the coordinate offset and window width and height of each text to be identified relative to each feature in the image feature vector. Based on the position of the feature, the coordinate offset, and the window width and height, calculate the window information corresponding to each text to be identified relative to each feature in the image feature vector. The window information corresponding to each key feature in the image feature vector is used as the text attention region.

3. The method according to claim 2, characterized in that, The calculation of the probability that each feature in the image feature vector belongs to a text region includes: The image feature vector is input into the classification linear layer to obtain the probability that each feature in the image feature vector output by the classification linear layer belongs to the text region; The calculation of the coordinate offset and window width and height of each text to be recognized relative to the position of each feature in the image feature vector includes: The image feature vector is input into a local window converter composed of a multilayer perceptron to obtain the coordinate offset and window width and height of each text to be recognized relative to the position of each feature in the image feature vector.

4. The method according to claim 1, characterized in that, The step of encoding the text interest region to obtain the query vector includes: The location query vector is obtained by performing location encoding on the text interest region; Text encoding is performed on the preset sequence start character to obtain the content query vector; The final query vector is obtained by combining the location query vector and the content query vector.

5. The method according to claim 1, characterized in that, The step of extracting image features corresponding to the text interest region to obtain a window feature vector includes: Extract the image features corresponding to the text interest region to obtain the initial window features; Using uniform width and height dimensions, interpolation is performed on the initial window features corresponding to each text attention region to obtain the window feature vector corresponding to each text attention region.

6. The method according to any one of claims 1 to 5, characterized in that, A pre-trained text recognition model is provided, which includes an encoder and a decoder. The encoder encodes the image to be recognized to obtain a query vector and a window feature vector. The decoder decodes the query vector and the window feature vector to obtain the text recognition result of the text to be recognized in the image.

7. The method according to claim 6, characterized in that, The encoder includes a feature extraction layer, a query vector construction layer, and a local window feature construction layer; The image to be identified is input into the feature extraction layer to extract image features and obtain an image feature vector; The image feature vector is input into the query vector construction layer to calculate the probability that each feature belongs to the text region, and the text attention region and query vector output by the query vector construction layer are obtained. The image feature vector and the text attention region are input into the local window feature construction layer to obtain the window feature vector output by the local window feature construction layer.

8. The method according to claim 6, characterized in that, The decoder includes a self-attention layer, a cross-attention layer, a feedforward neural network layer, and a classification layer; the step of decoding the text category and text position based on the query vector and the window feature vector to obtain the text recognition result of the text to be recognized in the image to be recognized includes: The query vector is encoded by text position and then input into the self-attention layer to obtain the self-attention query vector output by the self-attention layer; A linear transformation is performed on the window feature vector to obtain a value vector and a key vector. The value vector, the key vector, and the self-attention query vector are then input into the cross-attention layer to obtain the feature vector output by the cross-attention layer. The feature vector output by the cross-attention layer is input into the feedforward neural network layer for nonlinear transformation to obtain the feature vector output by the feedforward neural network layer. The feature vector output by the feedforward neural network layer is input into the classification layer to obtain the text category and text position of the text to be identified output by the classification layer, thereby obtaining the text recognition result of the text to be identified in the image to be identified.

9. The method according to claim 6, characterized in that, The training steps of the text recognition model include: Obtain image samples, which are labeled with text type tags and text location tags; The image sample is input into the encoder to be trained to obtain the probability prediction results of each feature belonging to the text region, the text attention region prediction results, the query vector, and the window feature vector output by the encoder to be trained. The query vector and the window feature vector are input into the decoder to be trained to obtain the text category prediction result output by the decoder to be trained; The probability prediction loss is calculated based on the probability prediction result and the text location label; the text region prediction loss is calculated based on the text attention region prediction result and the text location label; and the category prediction loss is calculated based on the text category prediction result and the text type label. The encoder to be trained is updated using the probability prediction loss and the text region prediction loss to obtain the trained encoder; the decoder to be trained is updated using the category prediction loss to obtain the trained decoder.

10. An electronic device, characterized in that, The electronic device includes a memory and a processor, the processor being configured to execute program instructions stored in the memory to implement the steps of the method as described in any one of claims 1-9.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program instructions that can be executed by a processor to implement the steps of the method as described in any one of claims 1-9.

Images