Text recognition method, device and storage medium

Abstract

The application discloses a text recognition method and device and a storage medium. The text recognition method comprises the following steps: performing image feature extraction on a to-be-recognized image to obtain an image feature vector; calculating the probability that each feature in the image feature vector belongs to a text region, and determining a text attention region based on the probability that each feature in the image feature vector belongs to the text region; performing position coding on the text attention region to obtain a query vector, and extracting the image features corresponding to the text attention region to obtain a window feature vector; and decoding a text category and a text position based on the query vector and the window feature vector to obtain a text recognition result of to-be-recognized text in the to-be-recognized image. The region position information that needs to be paid attention to and the image information of the region that needs to be paid attention to are integrated together and input into a decoder, so that the interaction between the query vector and irrelevant background information is reduced, the interference of other irrelevant region features in the image on text decoding is reduced, and the accuracy of the final output text recognition result is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a text recognition method, device and storage medium. BACKGROUND

[0002] Scene text recognition is a technology for locating and recognizing the text appearing in a scene image, facilitating further analysis of the content in the image.

[0003] Scene text recognition mainly includes two tasks of text positioning and recognition in an image. The commonly used recognition technology is implemented based on a deep learning method. Generally, a detection algorithm is used to output the bounding box of all texts in the image, and each text box region is cropped, rectified and enhanced, and then a character recognition algorithm is used to recognize each character in the text. However, the above method may introduce additional background noise in the cropping stage, affecting the accuracy of subsequent character recognition. SUMMARY

[0004] To solve the above technical problems, the present application at least provides a text recognition method, device and storage medium.

[0005] The first aspect of the present application provides a text recognition method, which comprises: acquiring an image to be recognized, performing image feature extraction on the image to be recognized to obtain an image feature vector; wherein the image to be recognized contains text to be recognized; calculating the probability that each feature in the image feature vector belongs to a text region, and determining a text attention region based on the probability that each feature in the image feature vector belongs to the text region; performing position coding on the text attention region to obtain a query vector, and extracting the image features corresponding to the text attention region to obtain a window feature vector; decoding the text category and the text position based on the query vector and the window feature vector to obtain the text recognition result of the text to be recognized in the image to be recognized.

[0006] In an embodiment, calculating the probability that each feature in the image feature vector belongs to the text region, and determining the text attention region based on the probability that each feature in the image feature vector belongs to the text region, comprises: calculating the probability that each feature in the image feature vector belongs to the text region, and selecting the features in the image feature vector that satisfy a preset condition to obtain key features; calculating the coordinate bias of each text to be recognized relative to the position of each feature in the image feature vector and the window width and height, and calculating the window information corresponding to each text to be recognized relative to each feature in the image feature vector based on the position of the feature, the coordinate bias and the window width and height; and taking the window information corresponding to each key feature in the image feature vector as the text attention region.

[0007] In an embodiment, the probability that each feature in the image feature vector belongs to the text region is calculated by inputting the image feature vector into a classification linear layer to obtain the probability that each feature in the image feature vector output by the classification linear layer belongs to the text region; and the coordinate bias and window width and height of each to-be-recognized text relative to the position of each feature in the image feature vector are calculated by inputting the image feature vector into a local window converter composed of multiple perception mechanisms to obtain the coordinate bias and window width and height of each to-be-recognized text relative to the position of each feature in the image feature vector output by the local window converter.

[0008] In an embodiment, the query vector is obtained by position encoding the text attention region, including: position encoding the text attention region to obtain a position query vector; text encoding a preset sequence start symbol to obtain a content query vector; and combining the position query vector and the content query vector to obtain the final query vector.

[0009] In an embodiment, the image feature corresponding to the text attention region is extracted to obtain a window feature vector, including: extracting the image feature corresponding to the text attention region to obtain an initial window feature; and performing interpolation processing on the initial window feature corresponding to each text attention region according to a unified width and height size to obtain a window feature vector corresponding to each text attention region.

[0010] In an embodiment, a text recognition model is pre-trained, and the text recognition model includes an encoder and a decoder. The encoder performs encoding processing on a to-be-recognized image to obtain a query vector and a window feature vector, and the decoder performs decoding processing on the query vector and the window feature vector to obtain a text recognition result of to-be-recognized text in the to-be-recognized image.

[0011] In an embodiment, the encoder includes a feature extraction layer, a query vector construction layer, and a local window feature construction layer. The to-be-recognized image is input into the feature extraction layer for image feature extraction to obtain an image feature vector. The image feature vector is input into the query vector construction layer to calculate the probability that each feature belongs to the text region, and the text attention region and the query vector output by the query vector construction layer are obtained. The image feature vector and the text attention region are input into the local window feature construction layer to obtain the window feature vector output by the local window feature construction layer.

[0012] In an embodiment, the decoder comprises a self-attention layer, a cross-attention layer, a feedforward neural network layer, and a classification layer; the text recognition result of the to-be-recognized text in the to-be-recognized image is obtained by decoding the text category and the text position based on the query vector and the window feature vector, comprising: inputting the query vector into the self-attention layer after text position encoding to obtain a self-attention query vector output by the self-attention layer; performing linear transformation on the window feature vector to obtain a value vector and a key vector, and inputting the value vector, the key vector, and the self-attention query vector into the cross-attention layer to obtain a feature vector output by the cross-attention layer; performing nonlinear transformation on the feature vector output by the cross-attention layer by inputting the feature vector into the feedforward neural network layer to obtain a feature vector output by the feedforward neural network layer; inputting the feature vector output by the feedforward neural network layer into the classification layer to obtain the text category and the text position of the to-be-recognized text output by the classification layer, and obtaining the text recognition result of the to-be-recognized text in the to-be-recognized image.

[0013] In an embodiment, the training step of the text recognition model comprises: obtaining an image sample, the image sample being labeled with a text type label and a text position label; inputting the image sample into the to-be-trained encoder to obtain a probability prediction result of each feature belonging to a text region, a text attention region prediction result, a query vector, and a window feature vector output by the to-be-trained encoder; inputting the query vector and the window feature vector into the to-be-trained decoder to obtain a text category prediction result output by the to-be-trained decoder; calculating a probability prediction loss based on the probability prediction result and the text position label, and calculating a text region prediction loss based on the text attention region prediction result and the text position label; calculating a category prediction loss based on the text category prediction result and the text type label; updating the to-be-trained encoder using the probability prediction loss and the text region prediction loss to obtain a trained encoder; and updating the to-be-trained decoder using the category prediction loss to obtain a trained decoder.

[0014] The second aspect of the present application provides a text recognition device, the device comprising: a feature extraction module, configured to obtain a to-be-recognized image, perform image feature extraction on the to-be-recognized image, and obtain an image feature vector; wherein the to-be-recognized image contains to-be-recognized text; a region determination module, configured to calculate a probability that each feature in the image feature vector belongs to a text region, and determine a text attention region based on the probability that each feature in the image feature vector belongs to the text region; a position and local feature encoding module, configured to perform position encoding on the text attention region to obtain a query vector, and extract image features corresponding to the text attention region to obtain a window feature vector; and a decoding module, configured to decode a text category and a text position based on the query vector and the window feature vector, and obtain a text recognition result of the to-be-recognized text in the to-be-recognized image.

[0015] The third aspect of the present application provides an electronic device, comprising a memory and a processor, the processor being configured to execute program instructions stored in the memory to implement the text recognition method described above.

[0016] The fourth aspect of the present application provides a computer readable storage medium having program instructions stored thereon, the program instructions being executed by a processor to implement the text recognition method described above.

[0017] The above scheme, by extracting image features from the image to be recognized, obtains an image feature vector; calculates the probability of each feature in the image feature vector belonging to the text region, determines the text attention region based on the probability of each feature in the image feature vector belonging to the text region; encodes the position of the text attention region to obtain a query vector, extracts the image features corresponding to the text attention region to obtain a window feature vector; based on the query vector and the window feature vector, the text category and the text position are decoded to obtain the text recognition result of the text to be recognized in the image to be recognized. The region position information that needs to be paid attention to and the image information of the region that needs to be paid attention to are integrated together and input into the decoder, which can reduce the interaction between the query vector and irrelevant background information, reduce the interference of other irrelevant region features in the image on the text decoding, and improve the accuracy of the final output text recognition result.

[0018] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, but not limiting the present application. BRIEF DESCRIPTION OF DRAWINGS

[0019] The drawings herein are incorporated into the specification and form part of the specification, which illustrate embodiments consistent with the present application, and together with the specification serve to explain the technical solutions of the present application.

[0020] Figure 1 is a schematic diagram of the implementation environment of the scheme shown by an exemplary embodiment of the present application;

[0021] Figure 2 is a flowchart of the text recognition method shown by an exemplary embodiment of the present application;

[0022] Figure 3 is a schematic diagram of the text recognition model shown by an exemplary embodiment of the present application;

[0023] Figure 4 is a schematic diagram of the query vector construction layer encoding to obtain a query vector shown by an exemplary embodiment of the present application;

[0024] Figure 5 is a schematic diagram of the local window feature construction layer to obtain a window feature vector shown by an exemplary embodiment of the present application;

[0025] Figure 6is a schematic diagram of a decoder performing decoding according to an example embodiment of the present application;

[0026] Figure 7 is a block diagram of a text recognition apparatus according to an example embodiment of the present application;

[0027] Figure 8 is a structural schematic diagram of an electronic device according to an example embodiment of the present application;

[0028] Figure 9 is a structural schematic diagram of a computer readable storage medium according to an example embodiment of the present application. DETAILED DESCRIPTION

[0029] The scheme of the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0030] In the following description, specific details such as specific system structures, interfaces, techniques, etc. are presented in order to provide a thorough understanding of the present application for the sake of explanation, but not for the purpose of limiting the present application.

[0031] The term "and / or" in the present document is merely a description of the association information of the associated objects, which means that there can be three relationships, for example, A and / or B can represent the following three cases: A exists alone, A and B exist together, and B exists alone. In addition, the character " / " in the present document generally represents an "or" relationship between the front and rear associated objects. In addition, "multiple" in the present document means two or more. In addition, the term "at least one" in the present document means any one of multiple or any combination of at least two of multiple, for example, including at least one of A, B and C can mean including any one or more elements selected from the set consisting of A, B and C.

[0032] The text recognition method provided by the embodiments of the present application will be described below.

[0033] Please refer to Figure 1 , Figure 1 is a schematic diagram of a scheme implementation environment according to an example embodiment of the present application. The scheme implementation environment can include a terminal 110 and a server 120, which are connected in communication with each other.

[0034] The number of terminals 110 can be one or more. The terminal 110 can be a camera, a smartphone, a tablet computer, a notebook computer, a desktop computer, a smart watch, etc., but is not limited thereto.

[0035] The server 120 can be a standalone physical server, a server cluster composed of multiple physical servers, or a distributed system, and can also be a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and basic cloud computing services such as big data and artificial intelligence platforms.

[0036] In one example, the server 120 can perform text recognition processing on the to-be-recognized image obtained from the terminal 110 to obtain a text recognition result. Of course, the server 120 can store the text recognition result locally, return it to the terminal 110, or transmit it to another terminal.

[0037] In one example, the terminal 110 has a client of a target application installed and running thereon, which can be an application providing a text recognition function. The terminal 110 performs text recognition processing on the to-be-recognized image by running the target application to obtain a text recognition result. The server 120 can be a background server of the target application, configured to provide background services for the client of the target application.

[0038] The text recognition method provided by the embodiments of the present application can be performed by the terminal 110, such as a client of a target application installed and running on the terminal 110, or by the server 120, or by the terminal 110 and the server 120 in cooperation, that is, part of the steps of the method are performed by the terminal 110 and the other part of the steps are performed by the server 120.

[0039] Referring to Figure 2 , Figure 2 is a flowchart of a text recognition method according to an example embodiment of the present application. The text recognition method can be applied to Figure 1 the implementation environment shown in the figure and is specifically performed by the server in the implementation environment. It should be understood that the method can also be applied to other example implementation environments and is specifically performed by devices in other implementation environments, and the present embodiment does not limit the implementation environment to which the method is applied.

[0040] As shown in Figure 2 , the text recognition method at least includes steps S210 to S240, which are described in detail as follows:

[0041] Step S210: obtaining a to-be-recognized image, performing image feature extraction on the to-be-recognized image to obtain an image feature vector; wherein the to-be-recognized image contains to-be-recognized text.

[0042] An image to be recognized is acquired, which is an image that needs to be recognized and can be an RGB image, a YUV image, an infrared image, a grayscale image, etc. The type of the image to be recognized is not limited in the present application.

[0043] The image to be recognized contains text to be recognized. The purpose of text recognition is to recognize the text type and / or text location of the text to be recognized in the image to be recognized.

[0044] Image feature extraction is performed on the image to be recognized to obtain an image feature vector.

[0045] The image feature extraction is used to extract information that can effectively represent the content or structure of the image from the image to be recognized, such as at least one of color features, texture features, shape features, spatial features, and abstract features generated by mathematical statistics or transformation of the image to be recognized.

[0046] For example, the image to be recognized can be subjected to image feature extraction based on a traditional mathematical method, such as using a Canny operator, a Sobel operator, and / or a Harris algorithm to perform image feature extraction on the image to be recognized to obtain an image feature vector.

[0047] For another example, the image to be recognized can be subjected to image feature extraction using a deep learning method, such as using a convolutional neural network (CNN) and / or a residual network (ResNet) that can be used for image feature extraction to perform image feature extraction on the image to be recognized to obtain an image feature vector.

[0048] For yet another example, the image to be recognized can be subjected to image feature extraction by combining a traditional mathematical method and a deep learning method to obtain an image feature vector.

[0049] It should be noted that the specific implementation of image feature extraction can be flexibly selected according to the actual application scenario, and the present application does not limit it.

[0050] Step S220: Calculate the probability of each feature in the image feature vector belonging to the text region, and determine the text attention region based on the probability of each feature in the image feature vector belonging to the text region.

[0051] The deep learning method uses a pre-trained neural network model, and the neural network model directly learns the mapping from features to probabilities end-to-end. The pre-trained neural network model is used to calculate the probability of each feature in the image feature vector belonging to the text region.

[0052] Specifically, the image feature vector is composed of multiple features, and a single feature can correspond to one or more pixel points in the to-be-identified image. The probability that each feature in the image feature vector belongs to the text region is identified by using a pre-trained neural network model.

[0053] The greater the probability that the feature belongs to the text region, the greater the probability that the pixel point corresponding to the feature is in the text region. Conversely, the smaller the probability that the feature belongs to the text region, the smaller the probability that the pixel point corresponding to the feature is in the text region. Therefore, the text attention region is determined according to the probability that each feature in the image feature vector belongs to the text region.

[0054] For example, the image region corresponding to the feature with a probability greater than a preset probability threshold is taken as the text attention region, or the image region corresponding to the top k (k≥1) features with the greatest probability is taken as the text attention region, which is not limited in the present application.

[0055] Step S230: Position encoding is performed on the text attention region to obtain a query vector, and image features corresponding to the text attention region are extracted to obtain a window feature vector.

[0056] The position encoding is performed on the text attention region to vectorize the position information of the text attention region.

[0057] The position encoding can be performed on the text attention region in a cosine position encoding or a sine position encoding manner, and the specific implementation manner of the position encoding is not limited in the present application.

[0058] In addition, the image features corresponding to the text attention region are extracted to obtain a window feature vector corresponding to the text attention region.

[0059] It should be noted that if the text attention region is multiple, image features can be extracted for each text attention region to obtain multiple window feature vectors.

[0060] Step S240: The text category and the text position are decoded based on the query vector and the window feature vector to obtain a text recognition result of the to-be-identified text in the to-be-identified image.

[0061] The query vector indicates the region position that needs to be focused in the to-be-identified image, and the window feature vector contains image information at the region position that needs to be focused. The text category and the text position are decoded by combining the query vector and the window feature vector, so that a more accurate text recognition result of the to-be-identified text in the to-be-identified image is obtained.

[0062] The application can reduce the interaction between the query vector and irrelevant background information, reduce the interference of other irrelevant region features in the image on text decoding, and improve the accuracy of the final output text recognition result by extracting a window feature vector, combining a query vector and the window feature vector for text decoding, and integrating the region position information that needs to be focused and the image information of the region that needs to be focused into the decoder.

[0063] Next, some embodiments of the application will be described in detail.

[0064] In some embodiments, the step of calculating the probability of each feature in the image feature vector belonging to the text region in step S220, determining the text focus region based on the probability of each feature in the image feature vector belonging to the text region, includes steps S221 to S223.

[0065] Step S221: Calculate the probability of each feature in the image feature vector belonging to the text region, and select the features in the image feature vector that meet the preset condition to obtain the key features.

[0066] For example, the image feature vector is input into the classification linear layer to obtain the probability of each feature in the image feature vector output by the classification linear layer belonging to the text region.

[0067] The linear layer (Linear Layer) is used for linear transformation of input data, and the classification linear layer of the application is used for linear transformation of each feature in the input image feature vector to map the probability of each feature belonging to the text region.

[0068] Select the features in the image feature vector that meet the preset condition to obtain the key features.

[0069] For example, the features with a probability greater than a preset probability threshold can be selected as the features that meet the preset condition to obtain the key features, or the top k (k≥1) features with the largest probability can also be selected as the features that meet the preset condition to obtain the key features.

[0070] Step S222: Calculate the coordinate bias and window width and height of each to-be-recognized text relative to the position of each feature in the image feature vector, and calculate the window information corresponding to each feature in the image feature vector for each to-be-recognized text based on the position of the feature, the coordinate bias and the window width and height.

[0071] Calculate the coordinate bias and window width and height of each to-be-recognized text relative to the position of each feature in the image feature vector.

[0072] The window width and height are used to describe the size of the text region, and the coordinate bias is used to describe the offset of the text region coordinates relative to the position of the feature. The text region coordinates can refer to the center coordinates of the text region.

[0073] For example, the image feature vector is input into a local window transformer composed of a multi-layer perception mechanism, and a coordinate bias of each feature in the image feature vector relative to the position of each feature to be recognized in the text is obtained.

[0074] The local window transformer can divide the input image feature vector into multiple local windows, and calculate the feature representation in each window independently.

[0075] The multi-layer perception mechanism (MLP) can learn a complex input-output mapping relationship through a nonlinear activation function, and is used to generate a position bias value to obtain a coordinate bias.

[0076] According to the position of the feature, the coordinate bias, and the window width and height, the window information corresponding to each feature in the image feature vector is calculated.

[0077] Specifically, the position of the feature and the coordinate bias calculated by the feature are added to obtain the window coordinates corresponding to the feature, and the window information is obtained by combining the window coordinates and the window width and height.

[0078] Step S223: The window information corresponding to each key feature in the image feature vector is taken as a text attention region.

[0079] The window information corresponding to each key feature is obtained, which is taken as a text attention region, and then the text attention region is positionally encoded to obtain a query vector.

[0080] In some embodiments, the text attention region is positionally encoded in step S230 to obtain a query vector, including steps S2311 to S2313.

[0081] Step S2311: The text attention region is positionally encoded to obtain a position query vector.

[0082] The position query vector is obtained by positionally encoding the text attention region, and provides text position information in the subsequent decoding process.

[0083] Step S2312: The preset sequence start symbol is textually encoded to obtain a content query vector.

[0084] For example, the preset sequence start symbol can be <sos>Of course, the sequence start symbol can also be represented in other ways, which is not limited in the present application.

[0085] Text encoding is performed on the preset sequence start symbol to obtain a content query vector.

[0086] Step S2313: Combining the position query vector and the content query vector, a final query vector is obtained.

[0087] By fusing the sequence start symbol information and the text position information of the text into the query vector, the sequence start symbol represents the beginning of sequence prediction, and serves as a trigger signal for the decoder to start generating the sequence, which can effectively guide the prediction of the text position and the text content in the decoding process.

[0088] In some embodiments, the step S230 of extracting the image features corresponding to the text attention regions to obtain the window feature vectors includes the following steps S2321-S2322.

[0089] Step S2321: Extracting image features corresponding to the text attention regions to obtain initial window features.

[0090] Step S2322: Interpolating the initial window features corresponding to each text attention region according to a unified width and height size to obtain a window feature vector corresponding to each text attention region, respectively.

[0091] The unified width and height size can be pre-set according to experience or can be flexibly calculated. For example, the width and height of each text attention region are counted to obtain the maximum width and the maximum height of each text attention region, and the maximum width and the maximum height are taken as the unified width and height size.

[0092] The initial window features corresponding to each text attention region are interpolated according to the unified width and height size, so that the width and height of the interpolated window feature vectors are the same.

[0093] The interpolation method includes but is not limited to bilinear interpolation, linear interpolation, and nearest neighbor interpolation.

[0094] Through interpolation, the region features of each text are unified to the same scale, so that the model gives the same attention to text instances of different sizes, which can effectively improve the recognition effect of small-scale text.

[0095] Then, the query vector and the window feature vector are input into the decoder to decode the text category and the text position, and a text recognition result of the text to be recognized in the image to be recognized is obtained.

[0096] In some embodiments, the text recognition model is pre-trained, and the text recognition model comprises an encoder and a decoder, the encoder encodes the to-be-recognized image to obtain a query vector and a window feature vector, and the decoder decodes the query vector and the window feature vector to obtain a text recognition result of the to-be-recognized image.

[0097] For example, refer to Figure 3 , Figure 3 is a schematic diagram of a text recognition model according to an example embodiment of the present application, as shown in Figure 3 , the encoder comprises a feature extraction layer, a query vector construction layer, and a local window feature construction layer, the to-be-recognized image is input into the encoder for encoding processing to obtain a query vector and a window feature vector, and the query vector and the window feature vector are input into the decoder for decoding processing to obtain a text recognition result.

[0098] For example, the feature extraction layer comprises a convolutional neural network (CNN) and a Transformer encoder, the to-be-recognized image is input into the feature extraction layer for image feature extraction to obtain an image feature vector.

[0099] Specifically, multi-scale features of the to-be-recognized image are extracted by using a CNN backbone network, the channel numbers of the multi-scale feature maps are unified by convolution operation, then the multi-scale features are flattened and spliced to obtain an initial image feature , where B represents a batch size, S represents a sum of width and height products of the plurality of scale feature maps, and C represents a channel number of the features. The CNN backbone network can be any network structure with image feature extraction capability, such as a ResNet network, a SwinTransformer (Shifted Window Transformer) network, a ViT (Vision Transformer) network, etc.

[0100] The initial image feature F is then sent into a Transformer encoder, the Transformer encoder learns global information of the image, adjusts attention degrees of features at different positions in the initial image feature F, strengthens key features, and weakens irrelevant features to obtain a final image feature vector .

[0101] Then, the image feature vector is input into the query vector construction layer to calculate probabilities that each feature belongs to a text region, and a text attention region and a query vector output by the query vector construction layer are obtained.

[0102] For example, refer to Figure 4 , Figure 4 is a schematic diagram of a query vector construction layer according to an example embodiment of the present application, as shown in Figure 4 The image feature vector is input into a classification linear layer, and the classification linear layer outputs the image feature vector The probability of each feature belonging to the text region in the image feature vector The image feature vector is input into a classification linear layer, and the classification linear layer outputs the image feature vector The index of the feature with the highest probability is obtained, and the image feature vector is obtained The image feature vector is input into a local window transformer composed of multiple perception mechanisms, and the local window transformer outputs the coordinate bias of each to-be-recognized text relative to the position of each feature And the window width and height (w, h), to obtain The position of each feature plus the corresponding coordinate bias is the text region center coordinate of the to-be-recognized text predicted by the feature, thereby obtaining the corresponding window information According to the index of the feature with the highest probability, the window information corresponding to the top k features is obtained The image feature vector is input into a classification linear layer, and the classification linear layer outputs the image feature vector After the cosine position encoding, the image feature vector is input into a position encoder composed of an MLP for encoding to obtain a position query vector .

[0103] And a preset sequence starting symbol <sos>The corresponding word vector is repeated k times to obtain a content query vector .

[0104] The position query vector and the content query vector are added to obtain a final query vector .

[0105] In addition, the image feature vector and the text attention region are input into the local window feature construction layer to obtain a window feature vector output by the local window feature construction layer.

[0106] For example, refer to Figure 5 , Figure 5 is a schematic diagram of the window feature vector obtained by the local window feature construction layer according to an exemplary embodiment of the present application, as shown in Figure 5 , according to the window information corresponding to the first k features , the shape of the local window corresponding to the k query vectors is obtained, and the initial window feature is obtained from the image feature vector . Each initial window feature is interpolated to , where g is the width and height of the interpolated window, so as to unify the width and height dimensions of all window features, to obtain a window feature vector. Each window feature vector is then flattened to obtain a feature sequence with a length of g x g.

[0107] The query vector of the i-th text attention region and the window feature vector are input into the decoder pair for decoding processing, where 1≤i≤k.

[0108] For example, the decoder includes a self-attention layer, a cross-attention layer, a feedforward neural network layer, and a classification layer; wherein the self-attention layer, the cross-attention layer, and the feedforward neural network layer constitute a decoding layer, and the decoder can contain N decoding layers.

[0109] The text position encoded query vector is input into the self-attention layer to obtain a self-attention query vector output by the self-attention layer; the window feature vector is linearly transformed to obtain a value vector and a key vector, and the value vector, the key vector and the self-attention query vector are input into the cross-attention layer to obtain a feature vector output by the cross-attention layer; the feature vector output by the cross-attention layer is input into the feedforward neural network layer for nonlinear transformation to obtain a feature vector output by the feedforward neural network layer; and the feature vector output by the feedforward neural network layer is input into the classification layer to obtain a text category and a text position of the to-be-recognized text output by the classification layer, thereby obtaining a text recognition result of the to-be-recognized image.

[0110] For example, refer to Figure 6 , Figure 6 is a schematic diagram of a decoder performing decoding according to an example embodiment of the present application, as shown in Figure 6 , the query vector of the i-th text attention region is input into the decoder after text position encoding. The self-attention layer is specifically causal self-attention. The query vector after text position encoding is further input into the causal self-attention module, Add&Norm (residual connection and layer normalization) and then serves as a text query vector input into the cross-attention module, and the value vector and the key vector obtained by linear layer from the window feature vector are used to calculate cross-attention, thereby reducing the interference of irrelevant regions. The causal self-attention mask is used to shield the input after the current position in the sequence, so that the model can only access the sequence features of the current position and the previous positions when calculating the self-attention score.

[0111] The output of the cross-attention module is further input into the feedforward neural network (FFN) and Add&Norm to output the decoding layer, thereby obtaining , which is input into the next decoding layer, wherein L is the length of the text. The output of the last decoding layer is input into the classification layer, and the classification layer includes a linear layer and a Softmax (activation function) layer to obtain the text category confidence of the query vector corresponding to the decoding .

[0112] wherein, is the size of the feature dictionary. For example, the position coordinates of the text are discretely represented in the form of categories, i.e., the position coordinate value belongs to a certain category in "1"~" . represents the number of units after the image is discretized in width and height, and the total number of character categories in the text content is an "unknown" class and a background class are additionally added <sos>, comma <eos>Therefore, the feature dictionary size adopted by the present application The dictionary vector is updated together with the training of the text recognition network.

[0113] The output of the last layer of the decoder is post-processed to obtain the text recognition result of the image to be recognized, and the specific post-processing is as follows: the last dimension of the text category confidence C represents the confidence of the text category to which the text belongs at each position in the sequence, and the index corresponding to the maximum value of the confidence is the text category index at the position; the first two characters predicted by each query vector represent the coordinates (x, y) of the point of the position of the text to be recognized, and the subsequent characters represent the text category, and the text is decoded out <eos>i.e. all text predictions are finished. It is noted that the output characters x and y are in coordinates under the size of the image, and the coordinates can be converted back to the original image to be recognized according to the size of the original image to be recognized and the scale of the image.

[0114] In some embodiments, the training step of the text recognition model comprises: obtaining an image sample, the image sample being labeled with a text type label and a text position label; inputting the image sample into the encoder to be trained to obtain a probability prediction result of each feature output by the encoder to be trained belonging to a text region, a text attention region prediction result, a query vector and a window feature vector; inputting the query vector and the window feature vector into the decoder to be trained to obtain a text category prediction result output by the decoder to be trained; calculating a probability prediction loss based on the probability prediction result and the text position label, and calculating a text region prediction loss based on the text attention region prediction result and the text position label; calculating a category prediction loss based on the text category prediction result and the text type label; updating the encoder to be trained using the probability prediction loss and the text region prediction loss to obtain a trained encoder; and updating the decoder to be trained using the category prediction loss to obtain a trained decoder.

[0115] Obtain scene text image data containing text position and text type labels to obtain an image sample. The image sample is image data containing text, for example, the text position label includes the center point coordinates of each text in the image sample, the width and height of the minimum circumscribed rectangle, and the text type label includes the text content of each text in the image sample. For ease of training, a filler equal to the end-of-sentence symbol <pad>The text type label is filled to a uniform length L.

[0116] The image sample is input to the encoder to be trained, and each feature output by the encoder to be trained belongs to the probability prediction result of the text region, the text attention region prediction result, the query vector and the window feature vector. Wherein, the inference process of the encoder is similar to the inference process corresponding to each embodiment described above.

[0117] During training, after obtaining the probability prediction result of the text region, the text attention region prediction result, the query vector and the window feature vector, the probability prediction result, the text attention region prediction result, the text type label and the text position label corresponding to k features are used to match the text type label for each query vector. The query vector that matches is assigned a corresponding text type label, and the query vector that does not match is assigned a stop symbol <eos>As background text, the matched text contains L characters. Among them, the matching method can be Hungarian algorithm and the like.

[0118] Then, the L characters matched are mapped to the corresponding feature representation through the feature dictionary, and the query vector corresponding to the image sample is obtained. The feature representation of the L characters is spliced to obtain After text position coding, the input of the decoder is

[0119] After text position coding, the input of the decoder is After the causal self-attention module and Add&Norm, the text query vector is input into the cross-attention module, and the window feature vector is obtained after the linear layer. The key vector is calculated by cross-attention, which learns the visual features related to the text instance, effectively reduces the learning range of the query vector to the local area of the text, and reduces the interference of irrelevant areas on the overall learning process.

[0120] Similar to the inference process, the output of the cross-attention module is input into the feedforward neural network and Add&Norm after the feedforward neural network and Add&Norm, and the output of the last decoding layer is input into the linear layer and Softmax layer to obtain the Softmax layer output of the query vector After decoding, the text category confidence of the corresponding text category is obtained.

[0121] In the training process, the k groups of output sequences of the decoder are labeled and assigned to the text type label sequence of all position coordinates of the corresponding image sample. The assigned corresponding text type label sequence matches, and the assigned end-of-sequence token does not match. <eos>As background text. Among the inference process, the decoder predicts the next text category by looping the input from the next query vector and the text category decoded by the decoder in the previous loop until the end-of-sentence is predicted <eos>The text recognition result of the to-be-identified image is obtained.

[0122] A loss function is constructed, and the specific loss function includes: for the probability prediction result, a classification loss function is used to calculate the difference between the probability prediction result and the text position label, to obtain a probability prediction loss, wherein the classification loss function includes but is not limited to a Focal Loss classification loss function, a cross-entropy loss function, a Hinge Loss classification loss function, and the like, which are not limited by the present application.

[0123] For the text attention region prediction result, an L1 norm loss function and / or a Generalized Intersection over Union Loss (GIoU Loss) function and the like are used to calculate the difference between the text attention region prediction result and the text position label, to obtain a text region prediction loss.

[0124] For the text category prediction result, a maximum likelihood loss function and / or a mean square error loss function and the like are used to calculate the difference between the text category prediction result and the text type label, to obtain a text region prediction loss.

[0125] The encoder to be trained is updated according to the probability prediction loss and the text region prediction loss, to obtain a trained encoder, the decoder to be trained is updated according to the category prediction loss, to obtain a trained decoder, and of course, the encoder to be trained can also be updated according to the category prediction loss, which is not limited by the present application. The gradient information of the decoding process of the decoder is transmitted back to the construction module of the query vector, to help adjust the window information of the text, so that the network adaptively learns how to more accurately locate the text region, and the features with higher relevance to the text are transmitted to the text decoder.

[0126] Figure 7 is a block diagram of a text recognition device according to an example embodiment of the present application. As shown in Figure 7 the example text recognition device 700 includes:

[0127] a feature extraction module 710 configured to obtain a to-be-identified image, perform image feature extraction on the to-be-identified image, and obtain an image feature vector; wherein the to-be-identified image contains to-be-identified text;

[0128] a region determination module 720 configured to calculate the probability that each feature in the image feature vector belongs to a text region, and determine a text attention region based on the probability that each feature in the image feature vector belongs to the text region;

[0129] a position and local feature encoding module 730 configured to perform position encoding on the text attention region to obtain a query vector, and extract image features corresponding to the text attention region to obtain a window feature vector;

[0130] The decoding module 740 is configured to decode the text category and the text position based on the query vector and the window feature vector, to obtain a text recognition result of the text to be recognized in the image to be recognized.

[0131] It should be noted that the text recognition apparatus provided in the above embodiments and the text recognition method provided in the above embodiments belong to the same concept, and the specific manner in which each module and unit performs operations has been described in detail in the method embodiments, which will not be described here. The text recognition apparatus provided in the above embodiments can be used in actual applications, and the above functions can be completed by different functional modules according to needs, that is, the internal structure of the apparatus is divided into different functional modules to complete all or part of the functions described above, which is not limited here.

[0132] Please refer to Figure 8 , Figure 8 is a structural schematic diagram of an electronic device according to an example embodiment of the present application. The electronic device 800 includes a memory 801 and a processor 802, and the processor 802 is configured to execute program instructions stored in the memory 801 to implement the steps in any of the above text recognition method embodiments. In a specific implementation scenario, the electronic device 800 can include but is not limited to a microcomputer, a server, and in addition, the electronic device 800 can also include a notebook computer, a tablet computer and other mobile devices, which are not limited here.

[0133] Specifically, the processor 802 is configured to control itself and the memory 801 to implement the steps in any of the above text recognition method embodiments. The processor 802 can also be referred to as a central processing unit (CPU). The processor 802 can be an integrated circuit chip having a processing capability of signals. The processor 802 can also be a general processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general processor can be a microprocessor or the processor can also be any conventional processor or the like. In addition, the processor 802 can be implemented by an integrated circuit chip together.

[0134] Please refer to Figure 9 , Figure 9 FIG. 9 is a structural schematic diagram of a computer readable storage medium according to an example embodiment of the present application. The computer readable storage medium 900 stores program instructions 910 capable of being executed by a processor, and the program instructions 910 are used to implement the steps in any of the text recognition method embodiments described above.

[0135] In some embodiments, the device provided by the embodiments of the present disclosure has functions or includes modules that can be used to perform the methods described in the above method embodiments, and the specific implementation can refer to the description of the above method embodiments. For brevity, details are not repeated here.

[0136] The above description of various embodiments tends to emphasize the differences between various embodiments, and the same or similar parts can be mutually referred to. For brevity, details are not repeated here.

[0137] In several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic; for example, the division of modules or units is only a logical function division, and actual implementation can have another division manner, for example, a unit or component can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the shown or discussed mutual units can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0138] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit. When the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to execute all or part of the steps of the methods in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk, and various media that can store program codes.< / eos> < / eos> < / eos> < / pad> < / eos> < / eos> < / sos> < / sos> < / sos>

Claims

1. A text recognition method, characterized by, The method comprises: obtaining an image to be recognized, performing image feature extraction on the image to be recognized to obtain an image feature vector; wherein the image to be recognized contains text to be recognized; calculating the probability of each feature in the image feature vector belonging to a text region, selecting features in the image feature vector that satisfy a preset condition in terms of probability, and obtaining key features; calculating the coordinate bias and window width and height of each text to be recognized relative to the position of each feature in the image feature vector, and calculating the window information corresponding to each feature in the image feature vector for each text to be recognized based on the position of the feature, the coordinate bias, and the window width and height; taking the window information corresponding to each key feature in the image feature vector as a text attention region; performing position encoding on the text attention region to obtain a query vector, and extracting image features corresponding to the text attention region to obtain a window feature vector; decoding the text category and text position based on the query vector and the window feature vector to obtain the text recognition result of the text to be recognized in the image to be recognized.

2. The method of claim 1, wherein, The calculation of the probability of each feature in the image feature vector belonging to a text region comprises: inputting the image feature vector into a classification linear layer to obtain the probability of each feature in the image feature vector belonging to a text region output by the classification linear layer; The calculation of the coordinate bias and window width and height of each text to be recognized relative to the position of each feature in the image feature vector comprises: inputting the image feature vector into a local window converter composed of multiple perception mechanisms to obtain the coordinate bias and window width and height of each text to be recognized relative to the position of each feature in the image feature vector output by the local window converter.

3. The method of claim 1, wherein, The position encoding of the text attention region to obtain a query vector comprises: performing position encoding on the text attention region to obtain a position query vector; performing text encoding on a preset sequence start symbol to obtain a content query vector; combining the position query vector and the content query vector to obtain a final query vector.

4. The method of claim 1, wherein, The extraction of the image features corresponding to the text attention region to obtain a window feature vector comprises: extracting image features corresponding to the text attention region to obtain initial window features; performing interpolation processing on the initial window features corresponding to each text attention region according to a unified width and height size to obtain window feature vectors corresponding to the text attention region respectively.

5. The method according to any one of claims 1 to 4, characterized in that, A text recognition model is pre-trained, the text recognition model comprises an encoder and a decoder, the encoder performs encoding processing on an image to be recognized to obtain a query vector and a window feature vector, and the decoder performs decoding processing on the query vector and the window feature vector to obtain a text recognition result of text to be recognized in the image to be recognized.

6. The method of claim 5, wherein, The encoder comprises a feature extraction layer, a query vector construction layer, and a local window feature construction layer; inputting the image to be recognized into the feature extraction layer to perform image feature extraction and obtain an image feature vector; inputting the image feature vector into the query vector construction layer to calculate probabilities that each feature belongs to a text region, to obtain a text attention region and a query vector output by the query vector construction layer; inputting the image feature vector and the text attention region into the local window feature construction layer to obtain a window feature vector output by the local window feature construction layer.

7. The method of claim 5, wherein, The decoder comprises a self-attention layer, a cross-attention layer, a feedforward neural network layer, and a classification layer; the query vector and the window feature vector are decoded to obtain a text recognition result of the text to be recognized in the image to be recognized, including: inputting the query vector into the self-attention layer after text position encoding to obtain a self-attention query vector output by the self-attention layer; performing linear transformation on the window feature vector to obtain a value vector and a key vector, inputting the value vector, the key vector, and the self-attention query vector into the cross-attention layer to obtain a feature vector output by the cross-attention layer; inputting the feature vector output by the cross-attention layer into the feedforward neural network layer to perform nonlinear transformation to obtain a feature vector output by the feedforward neural network layer; inputting the feature vector output by the feedforward neural network layer into the classification layer to obtain a text category and a text position of the text to be recognized output by the classification layer, to obtain the text recognition result of the text to be recognized in the image to be recognized.

8. The method of claim 5, wherein, The training step of the text recognition model comprises: obtaining an image sample, the image sample being marked with a text type label and a text position label; inputting the image sample into the encoder to be trained to obtain a probability prediction result that each feature belongs to a text region, a text attention region prediction result, a query vector, and a window feature vector output by the encoder to be trained; inputting the query vector and the window feature vector into the decoder to be trained to obtain a text category prediction result output by the decoder to be trained; calculating a probability prediction loss based on the probability prediction result and the text position label, calculating a text region prediction loss based on the text attention region prediction result and the text position label, and calculating a category prediction loss based on the text category prediction result and the text type label; updating the encoder to be trained using the probability prediction loss and the text region prediction loss to obtain a trained encoder, and updating the decoder to be trained using the category prediction loss to obtain a trained decoder.

9. An electronic device, comprising: The electronic device comprises a memory and a processor, and the processor is configured to execute program instructions stored in the memory to implement the steps in the method of any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program instructions that can be executed by a processor to implement the steps in the method of any one of claims 1-8.

Images