Text recognition method and device, electronic equipment, vehicle and storage medium

Abstract

The present application provides a kind of text recognition method, device, electronic equipment, vehicle and storage medium, the text recognition method is by obtaining target image to be identified;The target image to be identified is input into text recognition model, to output text recognition result according to text recognition model;The text recognition model includes feature extraction module, non-sequence text recognition module and sequence text recognition module, feature extraction module is used to extract the image feature in the target image to be identified, non-sequence text recognition module is used to obtain character position prediction probability graph according to image feature, sequence text recognition module is used to obtain text recognition result according to image feature and character position prediction probability graph, since non-sequence recognition module is not easy to identify wrong to single character;Sequence recognition module is better to predict the semantic relation of each character of the string to be identified, therefore, non-sequence recognition module and sequence recognition module are combined, and text recognition accuracy can be improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more particularly to a text recognition method, apparatus, electronic device, vehicle, and storage medium. Background Technology

[0002] Currently, character recognition technologies such as Optical Character Recognition (OCR) can be used to recognize text in images into text data that computers can understand. OCR technology refers to the process of analyzing and recognizing image files such as text documents and cards to obtain text and layout information, and it has wide applications in the field of image processing.

[0003] In scenarios such as autonomous driving and automated parking, character recognition technology is often used to identify text such as speed limit signs, license plate numbers, and parking space numbers. Existing text recognition models for autonomous driving and automated parking scenarios are either sequential or non-sequential recognition models. Sequential recognition models are good at extracting character association information between texts, but when recognizing new text (text that has not appeared in the training set), they may predict incorrect character association information, resulting in incorrect text string prediction. Non-sequential recognition models recognize characters based on features such as character shape, which is suitable for recognizing single characters, but is not good at extracting character association information of the entire text string. Summary of the Invention

[0004] This invention provides a text recognition method, device, electronic device, vehicle, and storage medium to address the shortcomings of existing text recognition models, which have low accuracy and cannot meet the needs of scenarios such as autonomous driving and automatic parking.

[0005] This invention provides a text recognition method, comprising:

[0006] Acquire the target image to be identified;

[0007] The target image to be identified is input into the text recognition model, and the text recognition result is output according to the text recognition model;

[0008] The text recognition model includes a feature extraction module, a non-sequential text recognition module, and a sequential text recognition module. The feature extraction module is used to extract image features from the target image to be recognized. The non-sequential text recognition module is used to obtain a character position prediction probability map based on the image features. The sequential text recognition module is used to obtain the text recognition result based on the image features and the character position prediction probability map.

[0009] According to a text recognition method provided by the present invention, the text recognition model further includes:

[0010] A feature attention module is used to weight and enhance the image features based on the location prediction probability to obtain enhanced features;

[0011] The sequential text recognition module is also used to output each individual character in the target image to be recognized based on the enhanced features.

[0012] According to a text recognition method provided by the present invention, the sequential text recognition module includes:

[0013] Flattening units are used to convert multidimensional augmented features into one-dimensional feature vectors;

[0014] LSTM units are used to combine the one-dimensional feature vectors in conjunction with contextual information;

[0015] The CTC loss unit is used to output individual characters based on the combined feature vector.

[0016] According to a text recognition method provided by the present invention, the text recognition model further includes:

[0017] A decoding module, connected to the feature extraction module, is used to perform upsampling and feature decoding operations on the image features;

[0018] The prediction module is connected to both the decoding module and the non-sequential text recognition module. It is used to obtain the height and width of the character box based on the output of the decoding module, so that the non-sequential text recognition module can obtain a character position prediction probability map based on the height and width of the character box.

[0019] According to a text recognition method provided by the present invention, the non-sequential text recognition module includes:

[0020] The character position prediction branch is used to predict the probability of each pixel in the image being within the character frame area based on the height and width of the character frame, and to obtain a character position prediction probability map.

[0021] An auxiliary training branch is used to output the character category prediction probability for each pixel when the character position prediction branch is being trained, and to use the character category as additional supervision information to train the character position prediction branch.

[0022] According to a text recognition method provided by the present invention, the method for calculating the probability of each pixel in the image falling within a character box region includes:

[0023] The probability value of pixel p within the character box region is d / max(w,h), where d is the distance from p to the character center point, which is obtained from the annotation information of the character annotation boxes in the model training set, w is the width of the character box, and h is the height of the character box.

[0024] The present invention also provides a text recognition device, comprising:

[0025] The acquisition module is used to acquire the target image to be identified;

[0026] An output module is used to input the target image to be recognized into a text recognition model, and output a text recognition result according to the text recognition model. The text recognition model includes a feature extraction module, a non-sequential text recognition module, and a sequential text recognition module. The feature extraction module is used to extract image features from the target image to be recognized. The non-sequential text recognition module is used to obtain a character position prediction probability map based on the image features. The sequential text recognition module is used to obtain the text recognition result based on the image features and the character position prediction probability map.

[0027] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the text recognition method described in any of the preceding claims.

[0028] The present invention also provides a vehicle including the electronic equipment described above.

[0029] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the text recognition method described in any of the preceding claims.

[0030] This invention provides a text recognition method, apparatus, electronic device, vehicle, and storage medium. The method involves acquiring a target image to be recognized; inputting the target image into a text recognition model to output a text recognition result; and the text recognition model includes a feature extraction module, a non-sequential text recognition module, and a sequential text recognition module. The feature extraction module extracts image features from the target image; the non-sequential text recognition module obtains a character position prediction probability map based on the image features; and the sequential text recognition module obtains the text recognition result based on the image features and the character position prediction probability map. Since the non-sequential recognition module is less prone to misrecognizing individual characters, and the sequential recognition module better predicts the semantic relationships between characters in the string to be recognized, combining the non-sequential and sequential recognition modules can improve the accuracy of text recognition. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0032] Figure 1 This is one of the flowcharts illustrating the text recognition method provided by the present invention;

[0033] Figure 2 This is the second flowchart illustrating the text recognition method provided by the present invention;

[0034] Figure 3 This is a schematic diagram of the structure of the text recognition device provided by the present invention;

[0035] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0037] Figure 1 A flowchart of the text recognition method provided in the embodiments of the present invention is shown below. Figure 1 As shown, the text recognition method provided in this embodiment of the invention includes:

[0038] Step 101: Obtain the target image to be identified;

[0039] Step 102: Input the target image to be recognized into the text recognition model to output the text recognition result according to the text recognition model; the text recognition model includes a feature extraction module, a non-sequential text recognition module and a sequential text recognition module. The feature extraction module is used to extract image features from the target image to be recognized. The non-sequential text recognition module is used to obtain a character position prediction probability map according to the image features. The sequential text recognition module is used to obtain the text recognition result according to the image features and the character position prediction probability map.

[0040] In this embodiment of the invention, the feature extraction module is used to perform downsampling and feature extraction operations on the image. The feature extraction module is the backbone network module, which is a network that extracts features. Its function is to extract information from the image for use by subsequent networks.

[0041] In this embodiment of the invention, the character position prediction probability map includes the probability that a pixel at each position in the image is a character, so that the specific position of each character can be determined based on the character position prediction probability map.

[0042] Traditional text recognition models for scenarios such as autonomous driving and automatic parking are either sequential or non-sequential. Sequential recognition models are good at extracting character association information between texts, but when recognizing new text, they may predict incorrect character association information, resulting in incorrect text string prediction. Non-sequential recognition models are based on features such as character shape, which are suitable for recognizing single characters, but are not good at extracting character association information of the entire text string.

[0043] This invention provides a text recognition method that acquires a target image to be recognized; inputs the target image into a text recognition model to output a text recognition result; the text recognition model includes a feature extraction module, a non-sequential text recognition module, and a sequential text recognition module. The feature extraction module is used to extract image features from the target image to be recognized; the non-sequential text recognition module is used to obtain a character position prediction probability map based on the image features; and the sequential text recognition module is used to obtain the text recognition result based on the image features and the character position prediction probability map. By combining the advantages of the non-sequential recognition model and the sequential recognition model, the text recognition accuracy is improved.

[0044] Based on any of the above embodiments, such as Figure 2 As shown, the text recognition model also includes:

[0045] The feature attention module is used to weight and enhance image features based on the location prediction probability to obtain enhanced features.

[0046] After downsampling, the non-sequential recognition module is input into the feature attention module. The feature attention module downsamples the position prediction probability map through a concatenation layer and a convolution layer, and then concatenates and convolves it with the image features to obtain enhanced features, thereby improving the prediction effect of the subsequent sequential text recognition module.

[0047] In this embodiment of the invention, the sequential text recognition module is further used for:

[0048] The enhanced features are used to output individual characters in the target image to be identified.

[0049] In this embodiment of the invention, the sequential text recognition module includes:

[0050] Flattening units are used to convert multidimensional augmented features into one-dimensional feature vectors;

[0051] In this embodiment of the invention, the flattening unit is used to "flatten" the input, that is, to reduce the multidimensional input to one dimension. It is commonly used in the transition from convolutional layers to fully connected layers. Flattening does not affect the batch size. It stretches a high-dimensional array along the x-axis or y-axis into a one-dimensional array.

[0052] LSTM (Long Short-Term Memory) units are used to combine the one-dimensional feature vectors in conjunction with contextual information;

[0053] In this embodiment of the invention, LSTM is a time-recurrent neural network designed to solve the long-term dependency problem of general recurrent neural networks. LSTM can solve the problem of poor learning performance for long-distance sentences.

[0054] The CTC (Connectionist Temporal Classification) loss unit is used to output individual characters based on the combined feature vector.

[0055] In this embodiment of the invention, the CTC loss can solve tasks with unequal input and output lengths for two reasons: First, CTC allows the model to output the same character in consecutive time segments, and by merging identical characters, the output of consecutive time segments becomes a single character; second, when there are repeated characters in the target output, such as the 'l' in 'hello', CTC introduces a blank character, and identical characters before and after the blank character are not merged. Because of the presence of repeated characters and blank characters, there may be multiple paths for the same output, so the purpose of CTC loss is to maximize the probability of all possible paths. The training of the neural network is guided by the CTC loss function. The output matrix of the neural network and the corresponding target (ground-truth, GT) text are input into the CTC loss function. It attempts all possible alignments of the GT text in the image and sums all the scores. If the sum of the alignment scores is high, the score of the GT text is high.

[0056] The text recognition model also includes:

[0057] The decoding module, connected to the feature extraction module, is used to upsample and decode image features; the decoding module transforms a fixed-length vector into the output target sequence.

[0058] The prediction module (head), connected to the non-sequential text recognition module, is used to obtain the height and width of the character box based on the output of the decoding module, so that the non-sequential text recognition module can obtain a character position prediction probability map based on the height and width of the character box.

[0059] The prediction module is the network that acquires the output content of the network. The prediction module uses the previously extracted features to make predictions.

[0060] In this embodiment of the invention, the non-sequential text recognition module includes:

[0061] The character position prediction branch is used to predict the probability of each pixel in the image being within the character box area based on the height and width of the character box.

[0062] An auxiliary training branch is used to output the character category prediction probability for each pixel when the character position prediction branch is being trained, and to train the character position prediction branch using the character category as additional supervision information.

[0063] In this embodiment of the invention, character categories include, but are not limited to, numbers, letters, and special symbols. Assisted training refers to adding new supervisory information (e.g., character category information) during training. The purpose is to enhance data diversity, improve training robustness and generalization, and ultimately strengthen the model's semantic features and robustness.

[0064] In this embodiment of the invention, the character position prediction branch is trained using a training loss to make its output more accurate. The training loss is, for example, focal loss. Focal loss addresses the problem of extreme imbalance between positive and negative samples in one-stage object detection. Imbalance means that in an image, there are typically only a dozen or so candidate boxes (positive samples) that match the target, while there are 10,000 to 100,000 unmatched candidate boxes (negative samples). These numerous negative samples not only fail to contribute to training the network but also drown out a small number of samples that are helpful for training. Focal loss is based on binary cross-entropy (CE). It is a dynamically scaled cross-entropy loss. Through a dynamic scaling factor, the weights of easily distinguishable samples during training can be dynamically reduced, thereby quickly focusing the attention on those difficult-to-distinguish samples, which may be positive or negative, but are all helpful for training the network.

[0065] In this embodiment of the invention, the method for calculating the probability of each pixel in an image falling within a character box region includes:

[0066] The probability value of pixel p within the character box region is d / max(w,h), where d is the distance from p to the character center point, the character center point is obtained from the annotation information of the character annotation boxes in the model training set, w is the width of the character box, and h is the height of the character box.

[0067] The text recognition method provided in this embodiment of the invention has the advantages of non-sequential recognition module, which is less likely to misidentify a single character; and sequential recognition module, which better predicts the semantic relationship between each character in the string to be recognized. Therefore, combining the two can improve the accuracy of text recognition.

[0068] The text recognition device provided by the present invention is described below. The text recognition device described below and the text recognition method described above can be referred to in correspondence.

[0069] Figure 3 This is a schematic diagram of a text recognition device provided in an embodiment of the present invention, such as... Figure 3 As shown, the text recognition device provided in this embodiment of the invention includes:

[0070] The acquisition module 301 is used to acquire the target image to be identified;

[0071] The output module 302 is used to input the target image to be recognized into the text recognition model, so as to output the text recognition result according to the text recognition model. The text recognition model includes a feature extraction module, a non-sequential text recognition module and a sequential text recognition module. The feature extraction module is used to extract image features from the target image to be recognized. The non-sequential text recognition module is used to obtain a character position prediction probability map according to the image features. The sequential text recognition module is used to obtain the text recognition result according to the image features and the character position prediction probability map.

[0072] In this embodiment of the invention, the sequential text recognition module includes:

[0073] Flattening units are used to convert multidimensional augmented features into one-dimensional feature vectors;

[0074] LSTM units are used to combine the one-dimensional feature vectors in conjunction with contextual information;

[0075] The CTC loss unit is used to output individual characters based on the combined feature vector.

[0076] In some embodiments of this invention, the sequence text recognition module is, for example, a CRNN (Convolutional Recurrent Neural Network) model, used for end-to-end recognition of text sequences of variable length. Instead of first segmenting individual characters, it transforms text recognition into a time-dependent sequence learning problem, i.e., image-based sequence recognition. This allows for the recognition of longer text sequences, effectively improving text recognition accuracy and making the model more robust. During prediction, the front end extracts features from the text image, fuses the feature vectors to extract contextual features of the character sequence, obtains the probability distribution of each feature column, and finally predicts the text sequence through a transcription layer.

[0077] In this embodiment of the invention, the non-sequential text recognition module includes:

[0078] The character position prediction branch is used to predict the probability of each pixel in the image being within the character box area based on the height and width of the character box.

[0079] An auxiliary training branch is used to output the center point of each character box during the training of the character position prediction branch. The character center point is used to supervise the probability of each pixel in the predicted image being within the character box region.

[0080] This invention provides a text recognition device that acquires a target image to be recognized; inputs the target image into a text recognition model to output a text recognition result based on the text recognition model; the text recognition model includes a feature extraction module, a non-sequential text recognition module, and a sequential text recognition module. The feature extraction module is used to extract image features from the target image to be recognized, the non-sequential text recognition module is used to obtain a character position prediction probability map based on the image features, and the sequential text recognition module is used to obtain a text recognition result based on the image features and the character position prediction probability map. By combining the advantages of the non-sequential recognition model and the sequential recognition model, the text recognition accuracy is improved.

[0081] Based on any of the above embodiments, the text recognition model further includes:

[0082] A feature attention module is used to weight and enhance the image features based on the location prediction probability to obtain enhanced features;

[0083] The sequential text recognition module is also used to output individual characters in the target image to be recognized based on the enhanced features.

[0084] The decoding module, connected to the feature extraction module, is used to perform upsampling and feature decoding operations on the image features;

[0085] The prediction module is connected to both the decoding module and the non-sequential text recognition module. It is used to obtain the height and width of the character box based on the output of the decoding module, so that the non-sequential text recognition module can obtain the character position prediction probability map based on the height and width of the character box.

[0086] The text recognition method provided in this embodiment of the invention has the advantages of non-sequential recognition module, which is less likely to misidentify a single character; and sequential recognition module, which better predicts the semantic relationship between each character in the string to be recognized. Therefore, combining the two can improve the accuracy of text recognition.

[0087] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4 As shown, the electronic device may include a processor 410, a communication interface 420, a memory 430, and a communication bus 440. The processor 410, communication interface 420, and memory 430 communicate with each other via the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a text recognition method. This method includes: acquiring a target image to be recognized; inputting the target image to be recognized into a text recognition model to output a text recognition result based on the text recognition model; the text recognition model includes a feature extraction module, a non-sequential text recognition module, and a sequential text recognition module. The feature extraction module is used to extract image features from the target image to be recognized; the non-sequential text recognition module is used to obtain a character position prediction probability map based on the image features; and the sequential text recognition module is used to obtain a text recognition result based on the image features and the character position prediction probability map.

[0088] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0089] On the other hand, the present invention also provides a vehicle including the electronic equipment described above.

[0090] Furthermore, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the text recognition method provided by the above methods. The method includes: acquiring a target image to be recognized; inputting the target image to be recognized into a text recognition model to output a text recognition result based on the text recognition model; the text recognition model includes a feature extraction module, a non-sequential text recognition module, and a sequential text recognition module, wherein the feature extraction module is used to extract image features from the target image to be recognized, the non-sequential text recognition module is used to obtain a character position prediction probability map based on the image features, and the sequential text recognition module is used to obtain a text recognition result based on the image features and the character position prediction probability map.

[0091] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0092] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0093] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A text recognition method, characterized in that, include: Acquire the target image to be identified; The target image to be identified is input into the text recognition model, and the text recognition result is output according to the text recognition model; The text recognition model includes a feature extraction module, a non-sequential text recognition module, and a sequential text recognition module. The feature extraction module is used to extract image features from the target image to be recognized. The non-sequential text recognition module is used to obtain a character position prediction probability map based on the image features. The sequential text recognition module is used to obtain the text recognition result based on the image features and the character position prediction probability map. The text recognition model also includes: The feature attention module is used to weight and enhance the image features according to the location prediction probability to obtain enhanced features. Specifically, the non-sequential recognition module structure is downsampled and then input into the feature attention module. The feature attention module downsamples the location prediction probability map through a concatenation layer and a convolution layer and then concatenates and convolves it with the image features to obtain enhanced features. The sequential text recognition module is also used to output each individual character in the target image to be recognized based on the enhanced features; The non-sequential text recognition module includes: The character position prediction branch is used to predict the probability of each pixel in the image being within the character box area based on the height and width of the character box, thus obtaining the character position prediction probability map. An auxiliary training branch is used to output the character category prediction probability for each pixel when the character position prediction branch is being trained, and to use the character category as additional supervision information to train the character position prediction branch.

2. The text recognition method according to claim 1, characterized in that, The sequential text recognition module includes: Flattening units are used to convert multidimensional augmented features into one-dimensional feature vectors; LSTM units are used to combine the one-dimensional feature vectors in conjunction with contextual information; The CTC loss unit is used to output individual characters based on the combined feature vector.

3. The text recognition method according to claim 1, characterized in that, The text recognition model also includes: A decoding module, connected to the feature extraction module, is used to perform upsampling and feature decoding operations on the image features; The prediction module is connected to both the decoding module and the non-sequential text recognition module. It is used to obtain the height and width of the character box based on the output of the decoding module, so that the non-sequential text recognition module can obtain a character position prediction probability map based on the height and width of the character box.

4. The text recognition method according to claim 1, characterized in that, The method for calculating the probability of each pixel in the image falling within the character box area includes: The probability value of pixel p within the character box region is d / max(w,h), where d is the distance from p to the character center point, which is obtained from the annotation information of the character annotation boxes in the model training set, w is the width of the character box, and h is the height of the character box.

5. A text recognition device, characterized in that, include: The acquisition module is used to acquire the target image to be identified; An output module is used to input the target image to be recognized into a text recognition model, and output a text recognition result according to the text recognition model. The text recognition model includes a feature extraction module, a non-sequential text recognition module, and a sequential text recognition module. The feature extraction module is used to extract image features from the target image to be recognized. The non-sequential text recognition module is used to obtain a character position prediction probability map based on the image features. The sequential text recognition module is used to obtain the text recognition result based on the image features and the character position prediction probability map. The text recognition model further includes a feature attention module, which is used to weight and enhance the image features according to the position prediction probability to obtain enhanced features. Specifically, the non-sequential recognition module structure is downsampled and input into the feature attention module. The feature attention module downsamples the position prediction probability map through a concatenation layer and a convolution layer, and then concatenates and convolves it with the image features to obtain enhanced features. The sequential text recognition module is also used to output each individual character in the target image to be recognized according to the enhanced features. The non-sequential text recognition module includes: a character position prediction branch, used to predict the probability of each pixel in the image being within the character frame region based on the height and width of the character frame, to obtain a character position prediction probability map; and an auxiliary training branch, used to output the character category prediction probability of each pixel when the character position prediction branch is being trained, and to use the character category as additional supervision information to train the character position prediction branch.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the text recognition method as described in any one of claims 1 to 4.

7. A vehicle, characterized in that, Including the electronic device as described in claim 6.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the text recognition method as described in any one of claims 1 to 4.

Images