Text recognition method, device, electronic device and storage medium
The cropped short text images are recognized through neural network models, and the context vectors are calculated using attention layer and loop layer, which solves the problems of slow text recognition speed and low accuracy in the prior art, and achieves fast and accurate long text images Identify.
Patent Information
- Application Number
- CN202210431773.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-22
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2042-04-22
AI Technical Summary
Existing text recognition methods are expensive to label when training models, and character-based methods may lead to problems such as missing recognition or multiple recognition, resulting in slow text recognition speed and low accuracy.
Using the neural network model, short text images are recognized and calculated by cropping text images into multiple short text images, and using multiple sequentially connected recognition modules, attention layers and loop layers in the pre-trained neural network model. Context vectors and decoding operations obtain the recognition results of text images.
It realizes fast and accurate recognition of long text images, reduces data annotation costs, and improves recognition speed and accuracy.
Smart Images

Figure CN114758331B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the technical field of text recognition, and in particular to a text recognition method, device, electronic device and storage medium. Background Art
[0002] With the development of image processing technology, image processing technology has been widely used in the field of text recognition. Machine learning and image processing methods are usually combined to automatically recognize text. For example, a large number of samples are obtained to train a machine learning model with recognition function, and then the text in the image is automatically recognized based on the trained recognition model. However, in the existing methods, the recognition model trained by the character-based method has a high annotation cost, and the recognition model trained by the sequence-based method may have problems of missed recognition or over-recognition, resulting in slow text recognition speed and low accuracy. Summary of the invention
[0003] In order to solve the above technical problems, the present disclosure provides a text recognition method, device, electronic device and storage medium, which can quickly and accurately recognize long text images in the form of short texts, and the recognition speed is also relatively fast.
[0004] According to one aspect of the present disclosure, a text recognition method is provided, comprising:
[0005] Acquire a text image, and crop the text image into multiple short text images;
[0006] The multiple short text images are input into a pre-trained neural network model, the multiple short text images are recognized by multiple recognition modules connected in sequence in the neural network model to obtain a first feature map and a first recognition result, a context vector is calculated according to the first feature map by using an attention layer in the neural network model, a decoding operation is performed based on the context vector and the first recognition result by using a recurrent layer in the neural network model to obtain a second recognition result corresponding to the text image.
[0007] According to another aspect of the present disclosure, there is provided a text recognition device, comprising:
[0008] An acquisition unit, used for acquiring a text image and cutting the text image into a plurality of short text images;
[0009] A recognition unit is used to input the multiple short text images into a pre-trained neural network model, recognize the multiple short text images through multiple recognition modules connected in sequence in the neural network model, obtain a first feature map and a first recognition result, use the attention layer in the neural network model to calculate a context vector according to the first feature map, use the recurrent layer in the neural network model to perform a decoding operation based on the context vector and the first recognition result, and obtain a second recognition result corresponding to the text image.
[0010] According to another aspect of the present disclosure, an electronic device is provided, the electronic device comprising: a processor; and a memory storing a program, wherein the program comprises instructions, and when the instructions are executed by the processor, the processor executes the above-mentioned text recognition method.
[0011] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, where the computer instructions are used to enable the computer to execute the above-mentioned text recognition method.
[0012] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the computer program implements the above-mentioned text recognition method.
[0013] The embodiments of the present disclosure provide a text recognition method, device, electronic device and storage medium. The method includes: obtaining a text image, and cutting the text image into multiple short text images; inputting the multiple short text images into a pre-trained neural network model, recognizing the multiple short text images through multiple recognition modules connected in sequence in the neural network model, obtaining a first feature map and a first recognition result, using the attention layer in the neural network model to calculate a context vector based on the first feature map, and using the loop layer in the neural network model to perform a decoding operation based on the context vector and the first recognition result to obtain a second recognition result corresponding to the text image. The method provided by the present disclosure can quickly and accurately recognize long text images in the form of short texts, and the recognition speed is also relatively fast. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.
[0015] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0016] Figure 1 A schematic diagram of an application scenario provided by an embodiment of the present disclosure;
[0017] Figure 2 A flowchart of a neural network model training method provided by an embodiment of the present disclosure;
[0018] Figure 3 A schematic diagram of the structure of a neural network model provided in an embodiment of the present disclosure;
[0019] Figure 4 A flowchart of a text recognition method provided by an embodiment of the present disclosure;
[0020] Figure 5 A schematic diagram of a text image provided by an embodiment of the present disclosure;
[0021] Figure 6 A structural schematic diagram of a text recognition device provided by an embodiment of the present disclosure;
[0022] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION
[0023] In order to more clearly understand the above-mentioned purposes, features and advantages of the present disclosure, the embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be interpreted as being limited to the embodiments described herein. Instead, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.
[0024] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.
[0025] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.
[0026] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".
[0027] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.
[0028] In view of the above problems, the embodiments of the present disclosure provide a text recognition method, which is specifically described through one or more of the following embodiments.
[0029] See also Figure 1 , Figure 1 A schematic diagram of an application scenario provided for an embodiment of the present disclosure, the model training method can be executed by the terminal 11 or the server 12. In a possible application scenario, the server 12 obtains a text image from the terminal 11 and recognizes the characters on the text image, and the text image can be obtained by photographing the terminal 11. Alternatively, the text image is an image obtained by the terminal 11 after performing image processing on a preset image, and the preset image can be obtained by photographing the terminal 11, or the preset image can be obtained by the terminal 11 from other devices. Here, no specific limitation is made to other devices.
[0030] In another application scenario, the server 12 obtains the text image by itself and recognizes the characters on the text image. The way in which the server 12 obtains the text image can be similar to the way in which the terminal 11 obtains the text image as described above, and will not be repeated here. It can be understood that the text recognition method provided by the embodiment of the present disclosure is not limited to the several possible scenarios described above. Before executing the text recognition method, it is necessary to first execute the training method of the neural network model to obtain the neural network model. The following takes the training of the neural network model by the server 12 as an example, and the training method of the neural network model is first described in detail. It can be understood that the neural network model training method is also applicable to the scenario in which the terminal 11 trains the neural network model.
[0031] The present disclosure provides a neural network model training method, which is specifically described through the following multiple embodiments. Figure 2 A flowchart of a neural network model training method provided in an embodiment of the present disclosure includes: Figure 2 The following steps S210 to S230 are shown:
[0032] S210: Acquire a training data set, where the training data set includes a large number of short text images and annotation information corresponding to each short text image in the large number of short text images.
[0033] It is understandable that the server obtains a large number of text images as training samples. A large number of text images include at least some short text images. Short text images can also be obtained by cropping long text images. The text in the text image can be straight text, curved text and oblique text. The form of the text in the text image is not limited. The characters included in the short text image are relatively short, that is, the number of characters included in the short text image is relatively small. For example, the short text image includes less than 10 characters, and the characters included in the short text image may also be incomplete. After obtaining a large number of short text images, each short text image is annotated, and the entire character sequence is annotated to obtain the annotation information corresponding to each short text image. A training data set of a neural network model is constructed based on the short text image and the annotation information. At the same time, after obtaining the annotation information, a dictionary corresponding to the neural network model is constructed based on the annotation information. The dictionary is used to determine the characters in the recognition process of the neural network model, wherein the dictionary is composed of the characters in all the annotation information and does not include repeated characters.
[0034] S220. Use the training data set to train multiple sequentially connected recognition modules in the neural network model to obtain a first training model, wherein the first training model includes multiple sequentially connected recognition modules that have been trained and an attention layer and a recurrent layer that have not yet been trained.
[0035] It can be understood that, based on the above S210, the training data set is input into the pre-constructed neural network model, and the multiple recognition modules connected in sequence of the neural network model are preferentially trained to obtain a first training model. That is, in one training, only the multiple recognition modules are preferentially trained through a large number of short text images, and the attention layer and the loop layer included in the neural network model are not trained to ensure the recognition accuracy of the neural network model. In this case, the loss value is calculated according to the recognition result output by the last recognition module among the multiple recognition modules and the annotation information, and the network parameters of the multiple recognition modules are updated according to the loss value to obtain the first training model that has been trained, wherein the first training model includes the multiple recognition modules that have been trained and the attention layer and the loop layer that have not yet been trained.
[0036] For example, see Figure 3 , Figure 3 A schematic diagram of the structure of a neural network model provided in an embodiment of the present disclosure, Figure 3It includes a plurality of recognition modules 310, an attention layer 330, a position encoding layer 320 and a circulation layer 340 connected in sequence; the plurality of recognition modules 310 connected in sequence include a first recognition module 311 and a second recognition module 312, the first recognition module 311 includes a first feature extraction layer and a first decoding layer, the first feature extraction layer uses a residual network as a backbone network, the residual network can be Resnet18, the residual network includes 4 convolution blocks, the convolution blocks are denoted as block blocks, each convolution block includes several convolution layers, the output of each block is the input of the next block block, The first feature extraction layer inputs a short text image and outputs a set of feature maps that are highly compressed to 1. The first feature extraction layer can also use other convolutional neural networks; the first decoding layer is used to decode a set of feature maps output by the first feature extraction layer to obtain an initial recognition result, an initial recognition result probability matrix, and the characters can be determined in the dictionary constructed above according to the probability matrix. Specifically, the first decoding layer includes a fully connected layer; the second recognition module 312 includes a second feature extraction layer and a second decoding layer. The second feature extraction layer is composed of two layers of bidirectional long short-term memory network (LSTM, Long The input of each time step of the second feature extraction layer is the feature map output by the first recognition module, and its hidden state is the recognition result output by the first decoding layer. The second feature extraction layer outputs a set of feature maps with the same dimension as the feature map output by the first recognition module; the attention layer 330 uses the feature map output by the second feature extraction layer as the key (Key) and value (Value), and uses the multiple position codes output by the position encoding layer 320 as the query parameter (Query) of each time step, calculates the similarity between the query parameter and the key, generates a score, and performs weighted average on the score and value to obtain the optimal representation feature corresponding to the character to be predicted at present, and the optimal representation feature is also the context vector; the position encoding layer 320 mainly includes a word embedding layer and a single-layer bidirectional long short-term memory network, and its main function is to provide a position code of a set dimension for each position in 0-L positions, where L refers to the maximum string length that can be recognized by the set neural network model. The multiple recognition modules 310 and the attention layer 330 are in series, and are in parallel with the position encoding layer 320; the recurrent layer 340 can be a gated recurrent unit (GatedRecurrent Unit, GRU), the recurrent layer 340 is used to perform decoding operations, the input of each time step of the recurrent layer 340 is the concatenation result of the context vector calculated by the attention layer 330 and the character predicted in the previous time step, the initial hidden state of the recurrent layer 340 is the recognition result output by the second recognition module, and the recurrent layer 340 outputs the final recognition result corresponding to the short text image.
[0037] Optionally, in the above S220, using the training data set to train a plurality of sequentially connected recognition modules in the neural network model to obtain a first training model specifically includes the following steps S221 to S223:
[0038] S221. Use the training data set to train a first recognition module among multiple recognition modules connected in sequence in the neural network model to obtain a recognition result corresponding to each short text image, and update the network parameters of the first recognition module according to the recognition result corresponding to each short text image and the annotation information corresponding to the short text image to obtain an updated first recognition module.
[0039] S222. Train the updated first recognition module using the training data set, extract features of multiple short text images in the training data set through multiple parallel first feature extraction layers in the updated first recognition module, decode features obtained by splicing features of the multiple short text images through the first decoding layer in the updated first recognition module, update the updated first recognition module based on the decoding result and information spliced from annotation information corresponding to the multiple short text images, and obtain a second training model, wherein the second training model includes the first recognition module that has been updated twice and other network layers in the neural network model that have not yet been trained.
[0040] S223. Use the training data set to train multiple sequentially connected recognition modules in the second training model to obtain a first training model.
[0041] It can be understood that, based on the above S210, the training of the neural network model is divided into four stages. The first stage: input the training data set into the neural network model to train the first recognition module among multiple recognition modules connected in sequence to obtain the recognition result corresponding to each short text image, and then update the network parameters of the first recognition module according to the recognition result corresponding to each short text image and the annotation information corresponding to the short text image to obtain the updated first recognition module, that is, first use the short text image to train the first recognition module to ensure the recognition accuracy of the short text image. The second stage: use the training data set to train the updated first recognition module. If the first recognition module is provided with a first feature extraction module, in this case, multiple short text images can be spliced into a long text image, for example, every 5 short text images are spliced into a long text image, and the annotation information corresponding to the 5 short text images also needs to be spliced together as the annotation information corresponding to the long text image. The long text image obtained by splicing the short text images is used to train the first recognition module. Alternatively, the first recognition module is provided with multiple parallel first feature extraction layers. In this case, multiple short text images can be directly input into the first feature extraction layer. For example, the first recognition module includes 5 parallel first feature extraction layers, and the structures of the 5 first feature extraction layers are the same. The above 5 short text images are respectively input into the 5 first feature extraction layers in sequence for feature extraction, and 5 groups of feature maps are output. The 5 groups of feature maps can also be The feature maps are spliced to obtain a group of feature maps, and then a group of feature maps spliced by features of multiple short text images or feature maps corresponding to each of the multiple short text images are decoded by the first decoding layer in the updated first recognition module. The loss value is calculated based on the decoding result and the information spliced by the annotation information corresponding to the multiple short text images. The loss function can be a CTC loss function. The updated first recognition module (the first recognition model output by the first stage) is updated by the loss value to obtain a second training model. By setting multiple parallel first feature extraction layers and sharing weights of the multiple parallel first feature extraction layers, rapid recognition training of long text images can be achieved through parallel input of short text images. The second training model includes the first recognition module that has completed two updates and other network layers in the neural network model that have not been trained. The other network layers refer to the second recognition module, the attention layer and the recurrent layer.The third stage: using the training data set to train multiple recognition modules connected in sequence in the second training model to obtain the first training model, that is, the training of the third stage adopts the splicing method involved in the second stage to jointly train all recognition modules, that is, the first recognition module and the second recognition module are trained through a large number of short text images, and at the same time, the training method for long text images in the second stage is adopted to splice the feature maps corresponding to the preset number of short text images output by the first recognition module to obtain a feature map corresponding to a long text image. Subsequently, the second recognition module is trained by the feature map corresponding to the spliced long text image to achieve the goal of long text image training, so as to improve the recognition accuracy of long text images. It is understandable that the spliced short text image can be a randomly acquired short text image, or it can be obtained by cropping a long text image. The specific method of acquiring the short text image is not limited, that is, there may be no semantic relationship between the multiple short text images spliced during the training process of the neural network model, that is, they may not be obtained by cropping a long text image.
[0042] S230: Train the first training model using the training data set to obtain a trained neural network model.
[0043] It can be understood that based on the above S220, the fourth stage of neural network model training is as follows: the first training model is trained using the training data set. At this time, the recognition module obtained through three stages of training in the first training model and the attention layer and the loop layer that have not been trained are jointly trained to obtain a neural network model with higher training recognition accuracy.
[0044] The disclosed embodiment provides a neural network model training method, which trains the neural network model by acquiring a large number of short text images. Short text images are easy to collect, and it is also relatively easy to annotate characters in the short text images, which can effectively reduce the difficulty of data collection and the cost of annotation, and further speed up the model training speed. The neural network model is trained in four stages, and the training of long text images is achieved by using short text images and subsequent splicing. Multiple recognition modules connected in sequence ensure the recognition accuracy of both short text images and long text images. In addition, the recognition results output by multiple recognition modules are iterated multiple times within the neural network model, which can further improve the model text recognition accuracy.
[0045] Based on the above embodiments, Figure 4 A flowchart of a text recognition method provided in an embodiment of the present disclosure is applied to the above-mentioned trained neural network model, specifically including the following: Figure 4 The following steps S610 to S420 are shown:
[0046] S410: Acquire a text image, and cut the text image into a plurality of short text images.
[0047] Specifically, the text recognition method can be executed by a terminal or a server. The terminal or the server can recognize characters in a text image through a neural network model. The execution subject of the training method of the neural network model and the execution subject of the text recognition method can be the same or different. For example, in an application scenario, Figure 1 As shown, the server 12 trains the neural network model, and the terminal 11 obtains the trained neural network model from the server 12. The terminal 11 recognizes the characters in the text image through the trained neural network model. The following description takes the terminal executing the text recognition method as an example.
[0048] It is understandable that the terminal obtains a text image to be recognized, and the text image to be recognized can be a single-line text image or a multi-line text image, which is not limited here. After the text image is obtained, the text image is cropped and cropped into multiple short text images. Specifically, the size of the cropped short text image can be set, that is, the image whose width and height are greater than the preset width and height is cropped. The preset width and height can be 32*160. For example, the width and height of the acquired text image is 256*256, which is greater than the preset width and height of 32*160. The text image is cropped according to the preset width and height of 32*160 to obtain multiple short text images. It is understandable that the multiple short text images obtained by cropping may contain different parts of the same character in different short text images, and may not include the character. In this case, all the cropped images can be directly input into the neural network model, and multiple short text images including at least part of the characters can also be input into the pre-trained neural network model.
[0049] For example, see Figure 5 , Figure 5 A schematic diagram of a text image is provided for an embodiment of the present disclosure, wherein text image 510 is an acquired text image, which is a multi-line text image. Text image 510 is cropped according to a preset width and height to obtain multiple short text images, which are recorded as short text image 520, short text image 530 and short text image 540, wherein short text image 540 includes a complete single line of text, short text image 530 and short text image 520 include partial text of the same line of text, and a line of text "BBBBBBBBB" in text image 510 is evenly divided into two parts, short text image 520 includes the upper half of the single line of text, and short text image 520 includes the lower half of the single line of text.
[0050] S420, inputting the multiple short text images into a pre-trained neural network model, recognizing the multiple short text images through multiple recognition modules connected in sequence in the neural network model to obtain a first feature map and a first recognition result, using an attention layer in the neural network model to calculate a context vector according to the first feature map, using a recurrent layer in the neural network model to perform a decoding operation based on the context vector and the first recognition result to obtain a second recognition result corresponding to the text image.
[0051] It can be understood that, based on the above S410, multiple short text images including at least one character are input into a pre-trained neural network model, and the multiple short text images are recognized by multiple recognition modules connected in sequence in the neural network model to obtain a first feature map and a first recognition result. Each recognition module in the multiple vaguely connected recognition modules will output a recognition result and a feature map, and the feature map of the recognition result is used as the input of the next recognition module, that is, multiple iterations are performed according to the recognition result to improve the recognition accuracy, wherein the feature map and recognition result output by the last recognition module in the multiple sequentially connected recognition modules are recorded as the first feature map and the first recognition result; after obtaining the first feature map and the first recognition result, the attention layer in the neural network model is used to calculate the score according to the first feature map and the multiple position codes output by the position coding layer, and then the context vector is calculated according to the score and the first feature map; after obtaining the context vector, the loop layer in the neural network model is used to perform a decoding operation based on the context vector and the first recognition result, that is, the last iteration is performed, and the loop layer will output the second recognition result corresponding to the text image, and the second recognition result is the final recognition result corresponding to the text image, which is also an accurate recognition result.
[0052] It is understandable that when there are two or more short text images among the multiple short text images obtained by segmentation, each of which includes different parts of the same line of text, the neural network model will fuse the recognition results corresponding to the two or more short text images before outputting the second recognition result to obtain the final second recognition result, for example, Figure 5 The medium-short text image 530 and the short text image 520 each correspond to a recognition result. One of the two recognition results can be randomly selected as the final recognition result of the line of text. The final recognition result of the line of text can also be determined by performing a fusion analysis based on the recognition results.
[0053] Optionally, the step of recognizing the multiple short text images by using multiple sequentially connected recognition modules in the neural network model to obtain a first feature map and a first recognition result in S420 specifically includes the following steps:
[0054] The first recognition module is used to perform preliminary recognition on the multiple short text images to obtain a second feature map and a third recognition result; and the second recognition module is used to perform recognition based on the second feature map and the third recognition result to obtain a first feature map and a first recognition result.
[0055] It can be understood that the following embodiments illustrate the recognition process of the recognition module in the neural network model by taking multiple recognition modules connected in sequence including two recognition modules as an example, wherein the first recognition module performs preliminary recognition on multiple short text images to obtain a second feature map and a third recognition result, wherein the second feature map can be obtained by splicing feature maps corresponding to multiple short text images, and the third recognition result can be understood as the initial recognition result; the second recognition module continues to perform recognition based on the second feature map and the third recognition result to obtain a first feature map and a first recognition result, wherein the first feature map can be understood as the feature map corresponding to the long text image, and the second recognition module is the second iteration based on the first recognition module, which is used to correct the third recognition result, wherein the third recognition result and the first recognition result are in the form of probability matrices, that is, except for the second recognition result output by the cyclic layer, the recognition results output by other recognition modules are in the form of probability matrices, which are used to perform multiple iterations within the neural network model to correct the recognition results.
[0056] Optionally, in the above S420, the first recognition module is used to preliminarily recognize the multiple short text images to obtain a second feature map and a third recognition result, which specifically includes the following steps S421 to S422:
[0057] S421. Perform a convolution operation on the multiple short text images through the first feature extraction layer to extract features of the multiple short text images to obtain a second feature map, wherein the second feature map is obtained by splicing feature maps corresponding to the multiple short text images in a cropping order, or the second feature map is composed of feature maps corresponding to the multiple short text images.
[0058] S422. Obtain a third recognition result according to the second feature map through the first decoding layer, wherein the third recognition result is an initial recognition result corresponding to the text image.
[0059] It can be understood that the processing flow inside the first recognition module is as follows: multiple short text images are convolved by multiple parallel first feature extraction layers to extract features of the multiple short text images and obtain a second feature map, wherein the second feature map can be a set of feature maps obtained by splicing feature maps corresponding to multiple short text images in a cropping order, or the second feature map is composed of feature maps corresponding to multiple short text images. In this case, the second feature map includes multiple sets of feature maps. After obtaining the second feature map, the second feature map is decoded by the first decoding layer to obtain a third recognition result, wherein the third recognition result is the initial recognition result corresponding to the text image. In this case, the initial recognition result is obtained by splicing a set of feature maps output by the first decoding layer based on multiple feature maps output by multiple parallel first feature extraction layers, which can be understood as the initial recognition result corresponding to the long text image.
[0060] Optionally, in the above S420, using the second recognition module to perform recognition based on the second feature map and the third recognition result to obtain a first feature map and a first recognition result specifically includes the following steps S423 to S424:
[0061] S423. Use the third recognition result as the hidden state of the second feature extraction layer, and extract the deep features of the second feature map based on the hidden state through the second feature extraction layer to obtain a first feature map, wherein the dimension of the first feature map is the same as the dimension of the second feature map.
[0062] S424. Recognize the first feature map through the second decoding layer to obtain a first recognition result, wherein the first recognition result is a depth recognition result corresponding to the text image.
[0063] It is understandable that the processing flow inside the second recognition module is as follows: the third recognition result is used as the hidden state of the second feature extraction layer, and then the second feature extraction layer extracts the deep features of the second feature map based on the hidden state, that is, further extracts the features of the characters in the second feature map, removes the background features, and obtains the first feature map, wherein the dimension of the first feature map is the same as the dimension of the second feature map. After obtaining the first feature map, the characters in the first feature map are recognized by the second decoding layer to obtain the first recognition result, wherein the first recognition result is the deep recognition result corresponding to the text image, and compared with the second recognition result, after two iterative corrections, the accuracy of the first recognition result is further improved. It is understandable that if the neural network model also includes a third recognition module, the input of the third recognition module is the output of the second recognition module, and the output of the third recognition module is also a feature map and a recognition result, and so on, until the last recognition module outputs a feature map and a recognition result, and each recognition module will output a feature map and a recognition result.
[0064] Optionally, the above S420 uses the attention layer in the neural network model to calculate the context vector according to the first feature map, which specifically includes the following steps S425 to S426:
[0065] S425. Provide a position code of a set dimension for each position in a preset number of positions through the position coding layer to obtain a preset number of position codes, wherein the preset number is determined according to a set maximum recognizable character string length.
[0066] S426. Utilize the attention layer in the neural network model to calculate an attention score according to the first feature map and the preset number of position codes, and obtain a context vector through the attention score and the first feature map.
[0067] It is understandable that before calculating the context vector, the attention layer will obtain a preset number of position codes output by the position coding layer in the neural network model. Specifically, the position coding layer will provide a position code of a set dimension for each of the preset number of positions to obtain a preset number of position codes, wherein the preset number is determined according to the set maximum recognizable string length. For example, the maximum recognizable string length is set to 20, and the preset number is 21, that is, from 0-20, there are 21 positions in total. The position coding layer will provide a position code of a preset dimension for each of the 21 positions, and the preset dimension can be the same as the dimension of the feature map output by the recognition module. After obtaining the preset number of position codes, the first feature map, and the first recognition result, the attention layer is used to calculate the attention score based on the first feature map and the preset number of position codes, and the context vector is obtained through the attention score and the first feature map.
[0068] Optionally, in the above S420, using the recurrent layer in the neural network model to perform a decoding operation based on the context vector and the first recognition result to obtain a second recognition result corresponding to the text image specifically includes the following steps S427 to S428:
[0069] S427: Use the first recognition result as the initial hidden state of the recurrent layer in the neural network model.
[0070] S428. For each time step of the loop layer, concatenate the context vector and the character predicted in the previous time step, and perform a decoding operation based on the initial hidden state and the concatenation result to obtain a second recognition result corresponding to the text image.
[0071] It can be understood that the processing flow inside the loop layer is as follows: the first recognition result is used as the initial hidden state of the loop layer in the neural network model. For each time step of the loop layer, the context vector output by the attention layer and the character predicted in the previous time step are concatenated. One character is predicted at each time step, and the input of the next time step is the concatenation result of the character predicted in the previous time step and the context vector. In one cycle, the loop layer will perform a decoding operation based on the initial hidden state and the concatenation result to obtain the second recognition result corresponding to the text image. If the loop layer performs multiple cycles, the next cycle will continue to perform the decoding operation based on the information obtained in the previous cycle. The number of cycles of the loop layer can be determined by the user according to user needs and is not limited here.
[0072] The embodiment of the present disclosure provides a text recognition method. After acquiring a text image, the text image is cropped into multiple short text images, and then the multiple short text images are input into a pre-trained neural network model. Multiple recognition modules connected in sequence in the neural network model recognize the multiple short text images in an aligned manner. The feature map and recognition result output by each recognition module are used as the input of the next recognition module. After multiple iterations, the first feature map and the first recognition result output by the last recognition module are obtained, and the initial recognition result can be obtained without considering the semantic information between the short text images; then, the attention layer in the neural network model is used to calculate the context vector according to the first feature map and the position coding output by the position coding layer, that is, to calculate the semantic information and the semantic association between the short text images; then, the recurrent layer in the neural network model is used to perform a decoding operation based on the context vector and the first recognition result to obtain a second recognition result, and the second recognition result is the final recognition result corresponding to the text image. The method provided by the present disclosure can realize the recognition of long text images based on short text images, and perform multiple iterations to correct the recognition results. While ensuring the recognition accuracy, the recognition speed is also relatively fast.
[0073] Based on the above embodiments, Figure 6 Schematic diagram of the structure of a text recognition device provided by an embodiment of the present disclosure. The text recognition device provided by an embodiment of the present disclosure can execute the processing flow provided by an embodiment of the model training method, such as Figure 6 As shown, the text recognition device 600 includes:
[0074] An acquisition unit 610 is used to acquire a text image and cut the text image into a plurality of short text images;
[0075] The recognition unit 620 is used to input the multiple short text images into a pre-trained neural network model, recognize the multiple short text images through multiple recognition modules connected in sequence in the neural network model, obtain a first feature map and a first recognition result, use the attention layer in the neural network model to calculate a context vector according to the first feature map, use the recurrent layer in the neural network model to perform a decoding operation based on the context vector and the first recognition result, and obtain a second recognition result corresponding to the text image.
[0076] Optionally, the plurality of sequentially connected identification modules in the identification unit 620 include a first identification module and a second identification module.
[0077] Optionally, the recognition unit 620 recognizes the multiple short text images by using multiple sequentially connected recognition modules in the neural network model to obtain a first feature map and a first recognition result, which is specifically used for:
[0078] Performing preliminary recognition on the multiple short text images by the first recognition module to obtain a second feature map and a third recognition result;
[0079] Using the second recognition module to perform recognition based on the second feature map and the third recognition result to obtain a first feature map and a first recognition result;
[0080] The third recognition result and the first recognition result are probability matrices.
[0081] Optionally, the first recognition module in the recognition unit 620 includes a first feature extraction layer and a first decoding layer.
[0082] Optionally, the recognition unit 620 performs preliminary recognition on the multiple short text images by using the first recognition module to obtain a second feature map and a third recognition result, which is specifically used for:
[0083] Performing a convolution operation on the multiple short text images through the first feature extraction layer to extract features of the multiple short text images to obtain a second feature map, wherein the second feature map is obtained by splicing feature maps corresponding to the multiple short text images in a cropping order, or the second feature map is composed of feature maps corresponding to the multiple short text images;
[0084] A third recognition result is obtained through the first decoding layer according to the second feature map, wherein the third recognition result is an initial recognition result corresponding to the text image.
[0085] Optionally, the second recognition module in the recognition unit 620 includes a second feature extraction layer and a second decoding layer; the using the second recognition module to perform recognition based on the second feature map and the third recognition result to obtain the first feature map and the first recognition result is specifically used for:
[0086] Using the third recognition result as a hidden state of the second feature extraction layer, and extracting deep features of the second feature map based on the hidden state through the second feature extraction layer to obtain a first feature map, wherein the dimension of the first feature map is the same as the dimension of the second feature map;
[0087] The first feature map is recognized through the second decoding layer to obtain a first recognition result, wherein the first recognition result is a depth recognition result corresponding to the text image.
[0088] Optionally, the recognition unit 620 uses the recurrent layer in the neural network model to perform a decoding operation based on the context vector and the first recognition result to obtain a second recognition result corresponding to the text image, which is specifically used for:
[0089] Using the first recognition result as the initial hidden state of the recurrent layer in the neural network model;
[0090] For each time step of the recurrent layer, the context vector and the character predicted in the previous time step are concatenated, and a decoding operation is performed according to the initial hidden state and the concatenated result to obtain a second recognition result corresponding to the text image.
[0091] Optionally, the apparatus 600 further includes a training unit, and the training unit is used in the training process of the neural network model, specifically for:
[0092] Acquire a training data set, wherein the training data set includes a large number of short text images and annotation information corresponding to each short text image in the large number of short text images;
[0093] Using the training data set to train multiple sequentially connected recognition modules in the neural network model to obtain a first training model, wherein the first training model includes the trained multiple sequentially connected recognition modules and the untrained attention layer and recurrent layer;
[0094] The first training model is trained using the training data set to obtain a trained neural network model.
[0095] Optionally, the training unit uses the training data set to train multiple sequentially connected recognition modules in the neural network model to obtain a first training model, which is specifically used for:
[0096] Using the training data set to train a first recognition module among a plurality of recognition modules connected in sequence in the neural network model, to obtain a recognition result corresponding to each short text image, and updating a network parameter of the first recognition module according to the recognition result corresponding to each short text image and the annotation information corresponding to the short text image, to obtain an updated first recognition module;
[0097] The updated first recognition module is trained using the training data set, features of multiple short text images in the training data set are extracted through multiple parallel first feature extraction layers in the updated first recognition module, and features obtained by splicing features of the multiple short text images are decoded through the first decoding layer in the updated first recognition module, and the updated first recognition module is updated based on the decoding result and information spliced from annotation information corresponding to the multiple short text images to obtain a second training model, wherein the second training model includes the first recognition module that has completed two updates and other network layers in the neural network model that have not yet been trained;
[0098] The training data set is used to train a plurality of sequentially connected recognition modules in the second training model to obtain a first training model.
[0099] Figure 6 The text recognition device of the illustrated embodiment can be used to execute the technical solution of the above-mentioned method embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.
[0100] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.
[0101] The implementation principle and technical effects of the device provided in this embodiment are the same as those of the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.
[0102] The exemplary embodiment of the present disclosure also provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program that can be executed by the at least one processor, and the computer program is used to cause the electronic device to perform the method according to the embodiment of the present disclosure when executed by the at least one processor.
[0103] The exemplary embodiments of the present disclosure further provide a computer program product, including a computer program, wherein when the computer program is executed by a processor of a computer, the computer is used to enable the computer to perform the method according to the embodiments of the present disclosure.
[0104] refer to Figure 7 , a block diagram of an electronic device 700 that can be used as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.
[0105] like Figure 7 As shown, the electronic device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.
[0106] A plurality of components in the electronic device 700 are connected to the I / O interface 705, including: an input unit 706, an output unit 707, a storage unit 708, and a communication unit 709. The input unit 706 may be any type of device capable of inputting information to the electronic device 700, and the input unit 706 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 707 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 704 may include, but is not limited to, a disk, an optical disk. The communication unit 709 allows the electronic device 700 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.
[0107] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above. For example, in some embodiments, the text recognition method or the training method of the recognition network may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 700 via the ROM 702 and / or the communication unit 709. In some embodiments, the computing unit 701 may be configured to perform the text recognition method or the training method of the recognition network by any other appropriate means (e.g., by means of firmware).
[0108] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0109] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0110] As used in this disclosure, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.
[0111] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0112] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.
[0113] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.
[0114] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A text recognition method, characterized in that: The method comprises: Acquire a text image, and crop the text image into multiple short text images; Input the multiple short text images into a pre-trained neural network model, recognize the multiple short text images through multiple recognition modules connected in sequence in the neural network model to obtain a first feature map and a first recognition result, use the attention layer in the neural network model to calculate a context vector based on the first feature map, use the recurrent layer in the neural network model to perform a decoding operation based on the context vector and the first recognition result to obtain a second recognition result corresponding to the text image; The plurality of sequentially connected recognition modules include a first recognition module and a second recognition module; the plurality of short text images are recognized by the plurality of sequentially connected recognition modules in the neural network model to obtain a first feature map and a first recognition result, including: Preliminarily recognizing the multiple short text images through the first recognition module to obtain a second feature map and a third recognition result, wherein the second feature map is obtained by splicing feature maps corresponding to the multiple short text images; Using the second recognition module to perform recognition based on the second feature map and the third recognition result to obtain a first feature map and a first recognition result, wherein the first feature map is a feature map corresponding to the text image; The third recognition result and the first recognition result are probability matrices, and the first recognition result is obtained by correcting the third recognition result.
2. The method according to claim 1, characterized in that The first recognition module includes a first feature extraction layer and a first decoding layer; the first recognition module performs preliminary recognition on the multiple short text images to obtain a second feature map and a third recognition result, including: Performing a convolution operation on the multiple short text images through the first feature extraction layer to extract features of the multiple short text images to obtain a second feature map, wherein the second feature map is obtained by splicing feature maps corresponding to the multiple short text images in a cropping order, or the second feature map is composed of feature maps corresponding to the multiple short text images; A third recognition result is obtained through the first decoding layer according to the second feature map, wherein the third recognition result is an initial recognition result corresponding to the text image.
3. The method according to claim 1, characterized in that The second recognition module includes a second feature extraction layer and a second decoding layer; the using the second recognition module to perform recognition based on the second feature map and the third recognition result to obtain a first feature map and a first recognition result includes: Using the third recognition result as a hidden state of the second feature extraction layer, and extracting deep features of the second feature map based on the hidden state through the second feature extraction layer to obtain a first feature map, wherein the dimension of the first feature map is the same as the dimension of the second feature map; The first feature map is recognized through the second decoding layer to obtain a first recognition result, wherein the first recognition result is a depth recognition result corresponding to the text image.
4. The method according to claim 1, characterized in that: The step of performing a decoding operation based on the context vector and the first recognition result using the recurrent layer in the neural network model to obtain a second recognition result corresponding to the text image includes: Using the first recognition result as the initial hidden state of the recurrent layer in the neural network model; For each time step of the recurrent layer, the context vector and the character predicted in the previous time step are concatenated, and a decoding operation is performed according to the initial hidden state and the concatenated result to obtain a second recognition result corresponding to the text image.
5. The method according to claim 1, characterized in that The training process of the neural network model includes: Acquire a training data set, wherein the training data set includes a large number of short text images and annotation information corresponding to each short text image in the large number of short text images; Using the training data set to train multiple sequentially connected recognition modules in the neural network model to obtain a first training model, wherein the first training model includes the trained multiple sequentially connected recognition modules and the untrained attention layer and recurrent layer; The first training model is trained using the training data set to obtain a trained neural network model.
6. The method according to claim 5, characterized in that The method of using the training data set to train a plurality of sequentially connected recognition modules in the neural network model to obtain a first training model includes: Using the training data set to train a first recognition module among a plurality of recognition modules connected in sequence in the neural network model, to obtain a recognition result corresponding to each short text image, and updating a network parameter of the first recognition module according to the recognition result corresponding to each short text image and the annotation information corresponding to the short text image, to obtain an updated first recognition module; The updated first recognition module is trained using the training data set, features of multiple short text images in the training data set are extracted through multiple parallel first feature extraction layers in the updated first recognition module, and features obtained by splicing features of the multiple short text images are decoded through the first decoding layer in the updated first recognition module, and the updated first recognition module is updated based on the decoding result and information spliced from annotation information corresponding to the multiple short text images to obtain a second training model, wherein the second training model includes the first recognition module that has completed two updates and other network layers in the neural network model that have not yet been trained; The training data set is used to train a plurality of sequentially connected recognition modules in the second training model to obtain a first training model.
7. A text recognition device, characterized in that: The device comprises: An acquisition unit, used for acquiring a text image and cutting the text image into a plurality of short text images; A recognition unit, configured to input the plurality of short text images into a pre-trained neural network model, recognize the plurality of short text images through a plurality of recognition modules connected in sequence in the neural network model, obtain a first feature map and a first recognition result, calculate a context vector based on the first feature map using an attention layer in the neural network model, and perform a decoding operation based on the context vector and the first recognition result using a recurrent layer in the neural network model to obtain a second recognition result corresponding to the text image; The plurality of sequentially connected identification modules include a first identification module and a second identification module; and the identification unit is used for: The first recognition module performs preliminary recognition on the multiple short text images to obtain a second feature map and a third recognition result, wherein the second feature map is obtained by splicing the feature maps corresponding to the multiple short text images; the second recognition module performs recognition based on the second feature map and the third recognition result to obtain a first feature map and a first recognition result, wherein the first feature map is the feature map corresponding to the text image; wherein the third recognition result and the first recognition result are probability matrices, and the first recognition result is obtained by correcting the third recognition result.
8. An electronic device, characterized in that: The electronic device comprises: Processor; and Memory for storing programs, The program includes instructions, which, when executed by the processor, enable the processor to perform the text recognition method according to any one of claims 1 to 6.
9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to enable the computer to execute the text recognition method according to any one of claims 1 to 6.
Citation Information
Patent Citations
Text recognition method, device and equipment
CN111695385A
Complex scene text recognition method and system, computer equipment and storage medium
CN112163596A
Text recognition method, text recognition model training method, storage medium and equipment
CN114092939A
Text recognition method, storage medium and computer terminal
CN114170468A
