Text recognition model generation method, text recognition method, device and storage medium

Abstract

A text recognition model generation method, a text recognition method, a device, and a storage medium are disclosed. The text recognition method includes: acquiring a text line image to be recognized; performing line text recognition on the text line image using a trained text line recognition model to obtain a text line recognition result; when it is determined that the confidence score of any character in the text line recognition result is less than a preset confidence score threshold or the energy of the character is greater than a preset energy threshold, cropping a character sub-image of a preset width centered on the corresponding character in the text line image; for each character sub-image, performing single-character detection and recognition using a trained single-character detection and recognition model to output the corresponding character recognition result; and combining the text line recognition result and the corresponding character recognition result to obtain the character recognition result of the text line image. This scheme can improve the accuracy and efficiency of text recognition for electronic devices with limited computing power.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of data processing technology, and in particular to a text recognition model generation method, a text recognition method, a device, and a storage medium. Background Technology

[0002] In recent years, with the continuous development and innovation of technologies in the field of Artificial Intelligence (AI), Optical Character Recognition (OCR) has achieved significant breakthroughs and has become a crucial foundational capability for digital transformation, intelligent upgrading, and integrated innovation in various vertical industries such as finance, transportation, logistics, education, and government. Due to the increasing maturity of the mobile internet and the accelerated development of the industrial internet, OCR service carriers and forms have become increasingly diversified. Offline OCR, which balances performance and efficiency, has become one of the future trends in technological development. Offline OCR has emerged as a new product form following public cloud application programming interfaces (APIs) and private deployments, expanding the boundaries of the broader OCR system. Compared to online services, it offers advantages such as low-cost deployment, zero data consumption, privacy protection, and WYSIWYG (What You See Is What You Get) functionality.

[0003] Driven by the reform of the "double reduction" policy in education and the gradual maturation of new-generation software and hardware technologies and industry chains, educational smart hardware, through the integration of diverse product functions such as AI algorithms, software, and content, helps students obtain personalized learning content and reasonably improve their learning methods, helps parents tutor and supervise their children's learning, and helps teachers improve teaching content and reduce teaching pressure.

[0004] According to iResearch's estimates, the market size of educational smart hardware reached 34.3 billion yuan in 2020 and is expected to approach 100 billion yuan by 2024, with emerging categories such as AI dictionary pens showing particularly strong growth. The AI ​​dictionary pen is a new generation of AI-powered dictionary pen designed for student users, enabling them to learn languages ​​through scanning and voice commands. It helps students solve problems related to listening, speaking, reading, writing, memorizing, translating, and looking up words. The core function of the AI ​​dictionary pen is scanning for word lookup. Because it uses an offline algorithm, the dictionary pen can be used normally in environments without internet access, greatly improving convenience and usability. Offline OCR is the entry point for extracting text information; if a recognition error occurs, the entire function becomes unusable. Furthermore, as it is a consumer product, user experience and hardware costs must be carefully considered. Therefore, achieving fast and highly accurate OCR recognition on electronic devices with limited computing power is a significant technical challenge facing the industry.

[0005] The content in the background section is merely technology known to the public and does not necessarily represent existing technology in this field. Summary of the Invention

[0006] In view of this, embodiments of this specification provide a text recognition model generation method, a text recognition method, a device, and a storage medium, which can improve the accuracy and efficiency of text recognition for electronic devices with limited computing power.

[0007] First, this specification provides a text recognition model generation method, including:

[0008] Obtain the training sets of text line images and character images respectively;

[0009] The text line image training set is input into a preset text line recognition model to train the text line recognition model; the character image training set is input into a preset single character detection and recognition model to train the single character detection and recognition model.

[0010] Obtain a test set of text line images, input the text line recognition model, and output the text line recognition result;

[0011] Evaluate the confidence and energy of each character in the text line recognition results;

[0012] When it is determined that the confidence level of any character in the text line recognition result is less than the preset confidence level threshold or the energy of the character is greater than the preset energy threshold, a character test sub-image of preset width is cropped with the corresponding character in the text line image as the center.

[0013] For each character test sub-image, input the single-character detection and recognition model, and output the corresponding character recognition result;

[0014] By combining the text line recognition results and the corresponding character recognition results, the character recognition test results of the text line image test set are obtained;

[0015] Based on whether the character recognition test results meet the preset performance evaluation indicators, it is determined whether to continue training the text line recognition model and the single character detection and recognition model until the character recognition test results meet the preset performance evaluation indicators.

[0016] Optionally, obtaining the text line image training set and the character image training set respectively includes:

[0017] The first text line image training set and the second text line image training set are mixed according to the first mixing ratio preset in the corresponding training batch to obtain the text line image training set; wherein, the first text line image training set is obtained by cropping the collected real text images; the second text line image training set is obtained by collecting font libraries of different fonts and synthesizing text lines according to the arrangement rules of the characters in the text line;

[0018] The first character image training set and the second character image training set are mixed according to the second mixing ratio preset in the corresponding training batch to obtain the character image training set; wherein, the first character image training set is obtained by cropping the collected real text images; the second character image training set is obtained by collecting font libraries of different fonts, synthesizing text images according to the text layout rules in the text line, and then cropping them.

[0019] Optionally, the first text line image training set and the second text line image training set are mixed according to a first mixing ratio preset in the corresponding training batch, and the first character image training set and the second character image training set are mixed according to a second mixing ratio preset in the corresponding training batch, so that the text line recognition model trained by the text line image training set and the single character detection and recognition model trained by the character image training set reach a preset generalization performance index threshold.

[0020] Optionally, the text line recognition model includes a convolutional recurrent neural network.

[0021] Optionally, the method further includes performing an image augmentation operation on the text line image training set and the character image training set.

[0022] Optionally, performing image augmentation on the text line image training set and the character image training set includes:

[0023] Image augmentation is performed on the text line image training set and character image training set according to the hyperparameters set in the corresponding training batches. The hyperparameters decay as the number of training batches increases.

[0024] This specification also provides a text recognition method, including:

[0025] Obtain the image of the text line to be recognized;

[0026] The trained text line recognition model is used to perform line text recognition on the text line image to obtain the text line recognition result;

[0027] Evaluate the confidence and energy of each character in the text line recognition results;

[0028] When it is determined that the confidence level of any character in the text line recognition result is less than a preset confidence threshold or the energy of the character is greater than a preset energy threshold, a character sub-image of a preset width is cropped with the corresponding character in the text line image as the center.

[0029] For each character sub-image, the trained single-character detection and recognition model is used to perform single-character detection and recognition, and the corresponding character recognition results are output.

[0030] By combining the text line recognition results and the corresponding character recognition results, the character recognition results of the text line image are obtained.

[0031] Optionally, the text line recognition model includes a convolutional recurrent neural network.

[0032] Optionally, the energy of each character in the text line recognition result is evaluated according to the following formula:

[0033]

[0034] Where X represents the image of the character, T represents the temperature hyperparameter, K represents the number of categories, and f is the text line recognition model network. i (x) represents the feature value corresponding to the i-th category out of K categories.

[0035] Optionally, before performing line text recognition on the text line image using the trained text line recognition model, the method further includes:

[0036] The image of the text line to be identified is subjected to image enhancement preprocessing.

[0037] This specification also provides a text recognition system, including:

[0038] The acquisition unit is adapted to acquire the image of the text line to be recognized;

[0039] The text line recognition unit is adapted to perform line text recognition on the text line image using a trained text line recognition model to obtain the text line recognition result;

[0040] The evaluation unit is adapted to evaluate the confidence and energy of each character in the text line recognition result;

[0041] The cropping unit is adapted to crop a character sub-image of a preset width centered on the corresponding character in the text line image when the evaluation unit determines that the confidence of any character in the text line recognition result is less than a preset confidence threshold or the energy of the character is greater than a preset energy threshold.

[0042] The character recognition unit is suitable for using a trained single-character detection and recognition model to perform single-character detection and recognition for each character sub-image, and outputs the corresponding character recognition results;

[0043] The recognition result output unit is adapted to combine the text line recognition result and the corresponding character recognition result to obtain and output the character recognition result of the text line image.

[0044] This specification also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, wherein when the processor runs the computer program, it performs the steps of the text recognition model generation method or the text recognition method described in any of the foregoing embodiments.

[0045] This specification also provides a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed, performs the steps of the text recognition model generation method or the text recognition method described in any of the foregoing embodiments.

[0046] The text recognition model generation method adopted in the embodiments of this specification involves acquiring a text line image training set and a character image training set, respectively. The text line image training set is used to train the text line recognition model, and the character detection and recognition model is trained using the character image training set. The model is then tested using a text line image test set. For the text line recognition result, the confidence and energy of each character in the result are evaluated. If the confidence of any character in the result is less than a preset confidence threshold or the energy of any character is greater than a preset energy threshold, a character test sub-image of a preset width is cropped centered on the corresponding character in the text line image. For each character test sub-image, the character detection and recognition model is input, and the corresponding character recognition result is output. The text line recognition result and the corresponding character recognition result are combined to obtain the character recognition test result of the text line image test set. Then, based on whether the character recognition test result meets a preset performance evaluation index, it is determined whether to continue training the text line recognition model and the character detection and recognition model until the character recognition test result meets the preset performance evaluation index. The text recognition model obtained by the above-mentioned text recognition model generation method introduces energy joint confidence to comprehensively judge the text line recognition results, which can more efficiently select the characters that need to be recognized and detected as individual characters; and, since it combines line recognition with the individual character detection and recognition model used for rare character recognition, the accuracy of character recognition can be improved by using the text recognition model.

[0047] The text recognition method described in this specification performs character recognition on a text line image. By evaluating the confidence and energy of each character in the text line recognition result, it can accurately distinguish between common and rare characters. When it is determined that the confidence of any character in the text line recognition result is less than a preset confidence threshold or the energy of the character is greater than a preset energy threshold, i.e., when the character is determined to be a rare character, a character sub-image of a preset width is cropped with the corresponding character in the text line image as the center. Then, for each character sub-image, a trained single-character detection and recognition model is used to perform single-character detection and recognition, and the corresponding character recognition result is output. Combining the text line recognition result and the corresponding character recognition result, the character recognition result of the text line image is obtained. On the one hand, since the entire recognition process introduces energy and confidence to comprehensively judge the text line recognition result, it can more efficiently select the characters that need to be recognized and detected as single characters. On the other hand, since it combines line recognition with the single-character detection and recognition model used for rare character recognition for comprehensive recognition, the use of the text recognition model for character recognition can improve the character recognition accuracy.

[0048] Furthermore, the text line image training set is obtained by cropping the collected real text images and the second text line image training set is obtained by collecting font libraries of different fonts and synthesizing text lines according to the arrangement rules of characters in the text lines, according to a first mixing ratio preset for the corresponding training batch; and the character image training set is obtained by cropping the first character image training set is obtained by cropping the second character image training set obtained by collecting font libraries of different fonts and synthesizing text images according to the arrangement rules of characters in the text lines, according to a second mixing ratio preset for the corresponding training batch. In other words, by adding the synthesized second text line image training set and the second character image training set during the training process, the text line image training set and the character image training set that meet the training requirements can be obtained more efficiently, thereby improving the training efficiency of the text recognition model.

[0049] Furthermore, by mixing the first text line image training set and the second text line image training set according to a preset first mixing ratio for the corresponding training batch, and mixing the first character image training set and the second character image training set according to a preset second mixing ratio for the corresponding training batch, the text line recognition model trained on the text line image training set and the single character detection and recognition model trained on the character image training set reach a preset generalization performance index threshold. This enables the trained text recognition model to more effectively express the features of real text image data and improves the generalization ability of the text recognition model.

[0050] By inputting a text line image into a convolutional recurrent neural network, the convolutional neural network can first extract the features of the text line image. Then, a bidirectional long short-term memory network is used to fuse the feature vectors to extract the contextual features of the character sequence. The probability distribution of each column of features is then obtained. Finally, the text sequence is predicted through a transcription layer. Since the bidirectional long short-term memory network and the transcription layer can learn the contextual features of the text line image, the accuracy of text recognition can be improved, and the robustness of the text recognition model can be enhanced.

[0051] Furthermore, by performing image augmentation operations on the text line image training set and the character image training set, the character recognition rate of the text recognition model in general scenarios can be improved.

[0052] Furthermore, by performing image augmentation operations on the text line image training set and character image training set according to the hyperparameters set in the corresponding training batches, since the hyperparameters decay with the increase of training batches, the fitting degree of the original text line image training set and character image training set can be gradually enhanced during the training process, thereby further improving the character recognition rate in general scenarios. Attached Figure Description

[0053] To more clearly illustrate the technical solutions of the embodiments of this specification, the drawings used in the description of the embodiments of this specification or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 A flowchart of a text recognition model generation method according to an embodiment of this specification is shown.

[0055] Figure 2 A flowchart of a text recognition method according to an embodiment of this specification is shown.

[0056] Figure 3 A schematic diagram illustrating a scenario example of a text recognition method according to an embodiment of this specification is shown.

[0057] Figure 3a It shows Figure 3 The image shown is an image of a line of text to be identified in an embodiment of this specification.

[0058] Figure 3b Examples of embodiments shown in this specification are illustrated. Figure 3a The text line recognition result is shown for the image of the text line to be recognized.

[0059] Figure 3c Examples of embodiments shown in this specification are illustrated. Figure 3bThe text line recognition results shown include the confidence score and energy of each character.

[0060] Figure 3d Examples of this specification are shown. Figure 3a The image shown is a partial character sub-image obtained from the text line image.

[0061] Figure 3e Examples of embodiments shown in this specification are illustrated. Figure 3d The corresponding character sub-image is input into a preset single-character detection and recognition model to obtain the character recognition result.

[0062] Figure 3f Examples of embodiments shown in this specification are combined with Figure 3b and Figure 3e The text recognition result is obtained from the character recognition result.

[0063] Figure 4 A schematic diagram of the structure of a text recognition system according to an embodiment of this specification is shown.

[0064] Figure 5 A schematic diagram of the structure of an electronic device according to an embodiment of this specification is shown. Detailed Implementation

[0065] As mentioned earlier, achieving fast and highly accurate OCR recognition on electronic devices with limited computing power is a technical challenge facing the industry, especially for texts with long-tail distribution characteristics and containing a large number of similar-looking characters. How to improve the accuracy and efficiency of text recognition is a technical problem that urgently needs to be solved.

[0066] To address the aforementioned problems, one aspect of the embodiments of this specification provides a text recognition model generation method. On one hand, by introducing energy joint confidence to comprehensively judge the text line recognition results, the characters that need to be recognized and detected as individual characters can be selected more efficiently. On the other hand, by combining line recognition with a single character detection and recognition model for rare character recognition, the accuracy of character recognition can be improved by using the text recognition model for character recognition.

[0067] To address the aforementioned problems, another aspect of the embodiments of this specification provides a text recognition method. On the one hand, since the entire recognition process incorporates energy joint confidence to comprehensively judge the text line recognition results, it can more efficiently select characters that need to be recognized and detected individually. On the other hand, since it combines line recognition with a single-character detection and recognition model for rare character recognition for comprehensive recognition, the use of the text recognition model for character recognition can improve character recognition accuracy.

[0068] To enable those skilled in the art to better understand and implement this invention, the concept, principles, and advantages of the embodiments of the present invention will be described in detail below through specific embodiments and in conjunction with specific application scenarios and accompanying drawings.

[0069] To ensure that the text recognition model achieves the expected performance metrics, it needs to be pre-tested on a training set until it meets the required performance specifications, at which point the desired text recognition model is generated. To help those skilled in the art better understand and implement this process, the following section provides a detailed explanation of the text recognition model generation process, along with specific application examples and accompanying diagrams.

[0070] Reference Figure 1 The flowchart shown illustrates a text recognition model generation method. In some embodiments of the present invention, the following steps can be used to generate a text recognition model for text character recognition:

[0071] S11, obtain the text line image training set and the character image training set respectively.

[0072] In specific implementation, the text line image training set can be obtained by identifying and cropping the regions of each line in each text image in the text image set. The character image training set can be obtained by identifying characters in the text line images and cropping them according to a preset width with each character as the center, or by first cropping each text image in the acquired text image set to obtain text line images, and then cropping the obtained text line images to obtain characters, or by directly recognizing characters in the acquired text image set and cropping them directly according to a preset region with the recognized characters as the center.

[0073] In practice, the training sets of text line images and character images can be obtained by collecting real text images, or they can be obtained by artificial synthesis.

[0074] In practical implementation, in order to balance the training efficiency of the text recognition model and the robustness of the training results, the collected real text line images and synthetic text line images can be mixed to obtain the text line image training set, and the collected real character images and artificially synthesized character images can be mixed to obtain the character image training set.

[0075] S12, input the text line image training set into a preset text line recognition model and train the text line recognition model; input the character image training set into a preset single character detection and recognition model and train the single character detection and recognition model.

[0076] In some embodiments of the present invention, a recurrent neural network (RNN) may be used. In specific implementations, the text line recognition model used may also be a combined model formed by combining RNN with other neural networks or algorithms, or an RNN that is further evolved, modified or extended based on RNN.

[0077] As an optional example, the text line recognition model can be a Convolutional Recurrent Neural Network (CRNN). CRNN is mainly used for end-to-end recognition of text sequences of variable length. Instead of segmenting individual characters first, it transforms text recognition into a time-dependent sequence learning problem, i.e., image-based sequence recognition.

[0078] Specifically, the entire network structure of CRNN consists of three parts: convolutional layers, recurrent layers, and transcription layers. As a concrete example, firstly, in the convolutional layers, a deep CNN can be used to extract features from the input text line image, obtaining a feature map; then, in the recurrent layers, a bidirectional RNN, more specifically a deep RNN such as a Bidirectional Long-Short Term Memory (BiLSTM) network, can be used to predict the feature sequence, learn each feature vector in the feature sequence, and output the predicted label (true value) distribution; finally, in the transcription layers, a Connectionist Temporal Classification (CTC) loss function can be used to transform the series of label distributions obtained from the recurrent layers into the final label sequence.

[0079] CTC is a loss function calculation method that does not require alignment of training samples. Its feature is that it solves the problem of missing characters in some positions by introducing the space character, and it can quickly calculate the gradient through recursion.

[0080] Specifically, for LSTM, there is a training set S = {(x1, z1), (x2, z2), ..., (x...} N , z N )}, where x is the feature map of the image obtained by CNN calculation, and z is the OCR character label corresponding to the image, which does not contain the blank character. Through gradient... Adjust the parameters w of the LSTM so that for input samples π∈B -1 When (z) is reached, p(l|x) reaches its maximum. Considering a single value in the y matrix of the CTC input (i.e., the LSTM output) in isolation... (Notice and The meanings are the same; both refer to π at time t. t =l k The probability of ) is:

[0081]

[0082] In the above formula, α t (l k )β t (l k The gradient is a constant calculated recursively, which can be obtained quickly at any time through recursion, thus allowing for rapid calculation of the gradient. Then you can train according to the gradient.

[0083] As another optional example, the text line recognition model can be a combination of a CNN and a Seq2Seq model structure and an attention model structure. The Seq2Seq model structure is an encoder-decoder structure, the basic idea of ​​which is to use two RNNs, one as an encoder and the other as a decoder. The attention model structure allows the c vector encoded by the encoder to be weighted and calculated with each line in the decoding process of the decoder, and the weights are adjusted to different c vectors in each decoding process.

[0084] It is understood that the embodiments in this specification do not limit the specific type of text line recognition model used, and those skilled in the art can choose to implement it according to actual needs.

[0085] In some embodiments of the present invention, a single character detection and recognition model can be used to first detect single characters, and then a dictionary can be used to identify the specific character. The embodiments of the present invention do not limit the specific type of the single character detection and recognition model used. As optional examples, the single character detection and recognition model can be a Region Proposal Network (RPN) structure model, or an Angled Region Proposal Network (RRPN) structure model, or an RCNN structure model, or a Fast-Recurrent Convolutional Neural Network (Fast-RCNN) structure model, or a combination thereof.

[0086] S13, obtain a text line image test set, input the text line recognition model, and output the text line recognition result.

[0087] S14, evaluate the confidence and energy of each character in the text line recognition result. If it is determined that the confidence of any character in the text line recognition result is less than a preset confidence threshold or the energy of the character is greater than a preset energy threshold, execute step S15.

[0088] S15, using the corresponding character in the text line image as the center, crop a character test sub-image of a preset width.

[0089] In practice, the confidence level can be calculated using the following formula:

[0090]

[0091] Where x represents the image of the character, K represents the number of categories, and y i Let f represent the feature value corresponding to the i-th category among K categories, and f be the text line recognition model network.

[0092] In practical implementation, the energy of each character in the text line recognition result can be evaluated according to the following formula:

[0093]

[0094] Where X represents the image of the character, T represents the temperature hyperparameter, K represents the number of categories, and f is the text line recognition model network. i (x) represents the feature value corresponding to the i-th category out of K categories.

[0095] In practice, rare characters can be defined as an out-of-distribution (OOD) detection problem, also known as an anomaly detection problem. By combining the energy and confidence of the characters for comprehensive judgment, rare characters and common characters have a clear distinction in energy distribution, which can more efficiently select the parts of the text line image that need to be detected and recognized as single characters.

[0096] In practice, since rare characters are generally not easy to recognize, their corresponding confidence levels are usually low. However, the energy distribution of rare characters is generally large, that is, the energy value of rare characters is large. Therefore, if the confidence level of a certain character is less than a preset confidence threshold or its energy is greater than a preset energy threshold, it can be regarded as a rare character and further input into the single character detection and recognition model to complete more accurate single character recognition.

[0097] The preset confidence threshold or energy threshold for the character can be set by analyzing samples after training the network and based on a unified empirical value. This specification does not limit the specific confidence threshold or energy threshold in the embodiments; it can be set according to factors such as the language type of the text, the text library and character set used for training, and the training network employed.

[0098] S16, For each character test sub-image, input the single-character detection and recognition model, and output the corresponding character recognition result.

[0099] S17, Combining the text line recognition results and the corresponding character recognition results, the character recognition test results of the text line image test set are obtained.

[0100] S18, determine whether the character recognition test result has reached the preset performance evaluation index. If yes, end the training; if no, continue to execute step S11 to continue training the text line recognition model and the single character detection and recognition model until the character recognition test result reaches the preset performance evaluation index.

[0101] In specific implementation, the text recognition model can be trained first through steps S11 and S12, and then tested through steps S13 to S17. Based on the obtained character recognition test results, it is determined whether the text recognition model has reached the preset performance evaluation index, and whether to continue training the text recognition model is selected based on the test evaluation results. The text recognition model includes a text line recognition model suitable for recognizing text lines and a single-character detection and recognition model suitable for recognizing individual characters. For the text line recognition results output by the text line recognition model, a comprehensive evaluation is performed using preset confidence thresholds and energy thresholds, which can more efficiently select characters that need to be recognized and detected as individual characters. Then, single-character recognition is performed using a single-character detection and recognition model for rare character recognition. Since the text recognition model combines the text line recognition model and the single-character detection and recognition model, the line recognition incorporates the character recognition results from the single-character detection and recognition model for rare character recognition. In summary, after the above text recognition model is trained, using the text recognition model to recognize characters in text line images can improve the accuracy and efficiency of character recognition.

[0102] In practice, due to the long-tail distribution characteristics of languages ​​such as Chinese and Japanese, and the presence of a large number of similar-looking characters, directly training text line recognition models such as CRNN to reach tens of thousands of categories of Chinese characters will bring some problems, such as accuracy issues. If there are many similar-looking characters, it will affect the recognition rate of common characters, and the recognition accuracy of rare characters in text line recognition models will be low. Through research and practice, the inventors discovered that in high-confidence intervals, it is difficult to effectively distinguish between rare and common characters. When relying solely on confidence for differentiation, if the threshold for classifying a character as a rare character is set too low, many common characters will be misclassified as rare characters and enter the single-character recognition branch, leading to increased computational load. If the confidence threshold is set too high, many rare characters will not be able to enter the single-character recognition branch, resulting in a very low rare character recognition rate. In the above embodiment, combining energy and confidence for rare character recognition, compared to single-confidence judgment, can improve the accuracy by 3% when the total number of characters is the same, for example, 20,000 characters. This solves the problem of low efficiency caused by directly training a text line recognition model with 20,000 classes, and can ensure the recognition accuracy of common characters, thus balancing recognition efficiency and recognition accuracy.

[0103] Furthermore, in practical implementation, the inventors discovered that directly training text line recognition models such as CRNN to cover tens of thousands of categories of Chinese characters can lead to other problems, such as data issues. Specifically, real data is expensive, while rare character corpora are scarce, and training with random corpora yields poor results.

[0104] To alleviate the above problems, in step S11, the first text line image training set and the second text line image training set can be mixed according to a first mixing ratio preset in the corresponding training batch to obtain the text line image training set; wherein, the first text line image training set is obtained by cropping the collected real text images; the second text line image training set is obtained by collecting font libraries of different fonts and synthesizing text lines according to the text line layout rules; similarly, the first character image training set and the second character image training set can be mixed according to a second mixing ratio preset in the corresponding training batch to obtain the character image training set; wherein, the first character image training set is obtained by cropping the collected real text images; the second character image training set is obtained by collecting font libraries of different fonts, synthesizing text images according to the text line layout rules, and then cropping them.

[0105] Due to the high cost of acquiring real data, the ratio of synthetic data to real data is typically 100:1 or even higher. If training is done using the same sampling ratio, the fitting effect of synthetic data is usually greater than that of real data, especially for difficult samples in real-world scenarios. This problem is more pronounced in small models with limited capacity. Therefore, in some embodiments of this invention, a batch ratio sampling strategy is adopted. Specifically, the first text line image training set and the second text line image training set can be mixed according to a first mixing ratio preset for the corresponding training batch, and the first character image training set and the second character image training set can be mixed according to a second mixing ratio preset for the corresponding training batch. This ensures that the text line recognition model trained on the text line image training set and the single character detection and recognition model trained on the character image training set reach a preset generalization performance threshold. As an optional example, the first mixing ratio can be, for example, 1:2, or 1:3, 1:4, etc.; similarly, the second mixing ratio can be 1:2, or 1:3, 1:4, etc. The above mixing ratio is only for illustrative purposes. In actual implementation, it is sufficient to ensure that the text line recognition model trained on the text line image training set and the single character detection and recognition model trained on the character image training set reach the preset generalization performance index threshold.

[0106] By employing the batch ratio sampling strategy described above, the sampling rate of real data can be increased, thereby improving the recognition accuracy in real-world scenarios and alleviating the training data problem to some extent.

[0107] To better generate the text recognition model, after executing step S11, image augmentation operations can be performed on the text line image training set and character image training set, and then step S12 can be executed to improve the robustness of the generated text recognition model.

[0108] Image augmentation involves making a series of random changes to the image training set to generate similar but different training samples, thereby expanding the size of the image set used for training. In specific implementations, image augmentation operations may include one or more of the following: random image cropping, random noise addition, affine transformation, projective transformation, and illumination contrast transformation. The specific method of image augmentation operation used in this embodiment is not limited.

[0109] Regularization techniques, such as data augmentation and dropout, have achieved great success in overfitting by adding noise, particularly in large neural networks. Dropout, in particular, refers to temporarily removing neural network units from the network with a certain probability during the training process.

[0110] However, these conventional regularization techniques can impair the performance of small neural networks because small models have limited capacity and are often underfitting. To address this issue, in some embodiments of this invention, a self-decaying image augmentation method can be employed. Specifically, image augmentation operations can be performed on the text line image training set and the character image training set according to the hyperparameters set for each training batch. These hyperparameters decay as the number of training batches increases. In specific implementations, the degree of image augmentation can be controlled by the hyperparameter ε. A larger ε indicates a greater difference between the augmented image and the original image. During training, ε can be decayed, similar to the learning rate. Self-decaying image augmentation can be seen as an alternative form of pre-training when ε is large. As ε gradually decreases, the fit of the original image data gradually increases, improving the character recognition rate in general scenarios.

[0111] By employing a self-decaying image augmentation method, the fitting degree of the original data can be gradually enhanced during the training process. A specific test verified that this ultimately improves the OCR recognition rate for general scenarios by about 2%.

[0112] Once trained, the above text recognition model can be used for text recognition. The following is a detailed introduction through specific embodiments and application scenarios.

[0113] Reference Figure 2 The flowchart of the text recognition method shown can be implemented in some embodiments of the present invention through the following steps:

[0114] S21, Obtain the image of the text line to be recognized.

[0115] S22, the trained text line recognition model is used to perform line text recognition on the text line image to obtain the text line recognition result.

[0116] As an optional example, the text line recognition model may include: CRNN. ​​As described in the foregoing embodiments, the text line recognition model may also employ other neural network models or algorithms. In this embodiment of the invention, no specific type or structure of the model used is limited.

[0117] S23, evaluate the confidence and energy of each character in the text line recognition result. If it is determined that the confidence of any character in the text line recognition result is less than a preset confidence threshold or the energy of the character is greater than a preset energy threshold, execute step S24.

[0118] As an optional example, the energy of each character in the text line recognition result can be evaluated using the following formula:

[0119]

[0120] Where X represents the image of the character, T represents the temperature hyperparameter, K represents the number of categories, f is the text line recognition model, such as CRNN, and fi(x) represents the feature value corresponding to the i-th category among the K categories.

[0121] S24, using the corresponding character in the text line image as the center, crop a character sub-image of a preset width.

[0122] In specific implementation, the preset width can be set based on the type of language and script to be recognized, image size, and text size, etc., and its specific value is not limited in this embodiment of the invention.

[0123] S25. For each character sub-image, the trained single-character detection and recognition model is used to perform single-character detection and recognition, and the corresponding character recognition results are output.

[0124] As described in the previous embodiments, a single character detection and recognition model can first detect single characters, and then use a dictionary to identify which specific character it is. This embodiment of the invention does not limit the specific type of single character detection and recognition model used. As an optional example, the single character detection and recognition model can be an RPN structure model, or an RRPN structure model, or an RCNN structure model, or a Fast-RCNN structure model, or a combination thereof.

[0125] S26, Combining the text line recognition result and the corresponding character recognition result, the character recognition result of the text line image is obtained and output.

[0126] By adopting the above embodiments, the text line recognition results are comprehensively judged by introducing energy joint confidence, which can more efficiently select the characters that need to be recognized and detected as individual characters. On the other hand, since the recognition is combined with the individual character detection and recognition model for rare character recognition, the accuracy of character recognition can be improved by using the text recognition model for character recognition.

[0127] In practical implementation, to improve the robustness of the text recognition model to general scenarios, before performing line text recognition on the text line image using the trained text line recognition model, image enhancement preprocessing can be performed on the text line image to be recognized. Specifically, various image enhancement processes such as median filtering, rotation, or size scaling can be applied to the text line image.

[0128] To enable those skilled in the art to better understand the embodiments, the following exemplary description is provided in conjunction with a specific application scenario, with reference to... Figure 3 and Figures 3a to 3f The diagram illustrates a scenario example of a text recognition method, for example, such as... Figure 3aThe image of the text line to be recognized is input into a preset text recognition model, such as a CRNN, which includes CNN layers and CTC layers. The output text line recognition result is as follows. Figure 3b As shown.

[0129] The confidence (Conf) and energy (Energy) of each character in the text line recognition result were evaluated and respectively satisfied as follows: Figure 3c The correspondence shown.

[0130] As an optional example, Energy can be calculated using the following formula:

[0131]

[0132] Where X represents the image of the character, T represents the temperature hyperparameter, K represents the number of categories, and f is the text line recognition model network. i (x) represents the feature value corresponding to the i-th category out of K categories.

[0133] Based on the judgment, refer to Figure 3c In the text line recognition results, the confidence level of character A01 is 0.64, which is lower than the preset confidence threshold Tc, while the energy value of character A02 is 400, which is greater than the preset energy threshold Te. Therefore, it is... Figure 3a Using the corresponding character in the text line image as the center, crop a character sub-image of a preset width, for example, to obtain two character sub-images as shown. Figure 3d The character sub-images B01 and B02 are shown below. These two sub-images are then input into a preset single-character detection and recognition model to obtain the corresponding character recognition results, as shown below. Figure 3e As shown, the confidence levels for the two are 0.93 and 0.96, respectively. Figure 3 As shown, if the preset recognition accuracy requirements are met, then combined with Figure 3b The text line recognition results shown are Figure 3e The character recognition results obtained from the single-character detection and recognition shown can be used to obtain the text recognition results of the text line image, as follows: Figure 3f As shown.

[0134] As shown above, combining the comprehensive judgment of character confidence and energy, and the comprehensive recognition of text lines by combining the text line recognition model with the single character recognition model, can improve text recognition accuracy. Furthermore, by comprehensively evaluating the confidence and energy of characters, single character detection and recognition that need to be further used for rare character recognition can be selected, thus avoiding the need to perform single character detection and recognition on a character-by-character basis in the text line image, thereby improving the overall recognition efficiency.

[0135] The embodiments in this specification also provide corresponding text recognition systems, such as... Figure 4The schematic diagram of the text recognition system shown indicates that the text recognition system 40 may include an acquisition unit 41, a text line recognition unit 42, an evaluation unit 43, a cropping unit 44, a character recognition unit 45, and a recognition result output unit 46, wherein:

[0136] The acquisition unit 41 is adapted to acquire the image of the text line to be identified;

[0137] The text line recognition unit 42 is adapted to perform line text recognition on the text line image using a trained text line recognition model to obtain the text line recognition result;

[0138] The evaluation unit 43 is adapted to evaluate the confidence and energy of each character in the text line recognition result;

[0139] The cropping unit 44 is adapted to crop a character sub-image of a preset width with the corresponding character in the text line image as the center when the evaluation unit determines that the confidence of any character in the text line recognition result is less than a preset confidence threshold or the energy of the character is greater than a preset energy threshold.

[0140] The character recognition unit 45 is adapted to use the trained single-character detection and recognition model to perform single-character detection and recognition for each character sub-image, and output the corresponding character recognition result;

[0141] The recognition result output unit 46 is adapted to combine the text line recognition result and the corresponding character recognition result to obtain and output the character recognition result of the text line image.

[0142] This specification also provides corresponding electronic devices in its embodiments, see below. Figure 5 The schematic diagram of the electronic device shown is illustrated in this specification. This embodiment also provides an electronic device 50, including a memory 51 and a processor 52. The memory 51 stores a computer program that can run on the processor 52. When the processor 52 runs the computer program, it can execute the steps of the text recognition model generation method or the steps of the text recognition method described in any of the foregoing embodiments. Specific steps can be found in the detailed description of the foregoing method embodiments.

[0143] Furthermore, in specific implementation, we will continue to refer to Figure 5 The electronic device 50 may also include a display module 53, which is adapted to output and display the text line recognition results.

[0144] In specific implementation, we will continue to refer to Figure 5The electronic device may further include an input interface 54, through which the user interacts to select a text line image to be recognized or to make some basic or personalized settings. As an optional example, the input interface 54 includes an optical scanning interface, which can directly acquire the text line image to be recognized; as another optional example, the input interface 54 includes a camera module, which can take pictures of the page or interface to be recognized to obtain the text line image.

[0145] In other embodiments, the image of the text line to be acquired can be obtained through communication interface 55, which can be a Bluetooth interface or other short-range communication interface.

[0146] In practice, the memory 51, processor 52, display module 53, input interface 54 and communication interface 55 can communicate with each other via bus 56.

[0147] As an optional example, the electronic device 50 can be a data processing device capable of offline processing, such as a dictionary pen. The processor 52 can be a single-core processor or a multi-core processor. It can be a general-purpose processor or a specially customized processor. There are no restrictions on the specific structure and implementation of the processor.

[0148] In practice, the dictionary pen is a product aimed at student users. It allows them to learn languages ​​through scanning or voice interaction, helping them solve problems related to listening, speaking, reading, writing, memorization, translation, and word lookup. The core function of the dictionary pen is scanning for words. Because it uses an offline algorithm, it can be used normally without a network connection, greatly improving its convenience and usability.

[0149] Offline OCR can serve as an entry point for extracting text information, i.e., as an input interface. If a recognition error occurs, the entire function will become unusable. At the same time, since it is a product aimed at end consumers, user experience and hardware costs need to be considered. Its data processing capabilities are usually limited. For example, it may use a single-core, dual-core, or quad-core low-power system-on-a-chip or processor.

[0150] Through the optimizations and improvements of this invention, and after testing and verification, the overall accuracy of the dictionary pen's uncommon character test set can be improved by more than 30%, with almost no impact on the recognition rate of common characters. Furthermore, the various improved designs for small-scale model training exemplified in the foregoing embodiments of this specification can meet the efficient and rapid user application needs in dictionary pen scenarios.

[0151] It is understood that the electronic device may also be a wearable device such as smart glasses or smartwatches, or a low-end mobile phone, or an Internet of Things device such as a portable scanner or smart speaker. The embodiments in this specification do not limit the specific type of electronic device that can apply the text recognition method and system described in the embodiments of this invention.

[0152] This specification also provides a computer-readable storage medium storing a computer program thereon, wherein the computer program, when running, executes the steps of the text recognition model generation method or the steps of the text recognition method described in any of the foregoing embodiments. Specific steps can be found in the foregoing embodiments and will not be repeated here.

[0153] In practice, the computer-readable storage medium can be any suitable readable storage medium such as optical disc, hard disk drive, solid-state drive, etc.

[0154] While the embodiments disclosed in this specification are as described above, the present invention is not limited thereto. Any person skilled in the art can make various modifications and alterations without departing from the spirit and scope of the invention; therefore, the scope of protection of the present invention should be determined by the scope defined in the claims.

Claims

1. A method for generating a text recognition model, wherein, include: Obtain the text line image training set and the character image training set respectively; The text line image training set is input into a preset text line recognition model to train the text line recognition model; the character image training set is input into a preset single character detection and recognition model to train the single character detection and recognition model. Obtain a test set of text line images, input the text line recognition model, and output the text line recognition result; Evaluate the confidence and energy of each character in the text line recognition results; When it is determined that the confidence level of any character in the text line recognition result is less than the preset confidence level threshold or the energy of the character is greater than the preset energy threshold, a character test sub-image of preset width is cropped with the corresponding character in the text line image as the center. For each character test sub-image, input the single-character detection and recognition model, and output the corresponding character recognition result; By combining the text line recognition results and the corresponding character recognition results, the character recognition test results of the text line image test set are obtained; Based on whether the character recognition test results meet the preset performance evaluation indicators, it is determined whether to continue training the text line recognition model and the single character detection and recognition model until the character recognition test results meet the preset performance evaluation indicators.

2. The method according to claim 1, wherein, The steps of obtaining the text line image training set and the character image training set respectively include: The first text line image training set and the second text line image training set are mixed according to the first mixing ratio preset in the corresponding training batch to obtain the text line image training set; wherein, the first text line image training set is obtained by cropping the collected real text images; the second text line image training set is obtained by collecting font libraries of different fonts and synthesizing text lines according to the arrangement rules of the characters in the text line; The first character image training set and the second character image training set are mixed according to the second mixing ratio preset in the corresponding training batch to obtain the character image training set; wherein, the first character image training set is obtained by cropping the collected real text images; the second character image training set is obtained by collecting font libraries of different fonts, synthesizing text images according to the text layout rules in the text line, and then cropping them.

3. The method according to claim 2, wherein, The first text line image training set and the second text line image training set are mixed according to the first mixing ratio preset in the corresponding training batch, and the first character image training set and the second character image training set are mixed according to the second mixing ratio preset in the corresponding training batch, so that the text line recognition model trained by the text line image training set and the single character detection and recognition model trained by the character image training set reach the preset generalization performance index threshold.

4. The method according to claim 1, wherein, The text line recognition model includes a convolutional recurrent neural network.

5. The method according to any one of claims 1-4, wherein, It also includes performing image augmentation operations on the text line image training set and the character image training set.

6. The method according to claim 5, wherein, The image augmentation operation performed on the text line image training set and the character image training set includes: Image augmentation is performed on the text line image training set and character image training set according to the hyperparameters set in the corresponding training batches. The hyperparameters decay as the number of training batches increases.

7. A text recognition method, wherein, include: Obtain the image of the text line to be recognized; The trained text line recognition model is used to perform line text recognition on the text line image to obtain the text line recognition result; Evaluate the confidence and energy of each character in the text line recognition results; When it is determined that the confidence of any character in the text line recognition result is less than a preset confidence threshold or the energy of the character is greater than a preset energy threshold, a character sub-image of a preset width is cropped with the corresponding character in the text line image as the center. For each character sub-image, the trained single-character detection and recognition model is used to perform single-character detection and recognition, and the corresponding character recognition results are output. By combining the text line recognition results and the corresponding character recognition results, the character recognition results of the text line image are obtained.

8. The method according to claim 7, wherein, The text line recognition model includes a convolutional recurrent neural network.

9. The method according to claim 7, wherein, The energy of each character in the text line recognition result is evaluated according to the following formula: Where X represents the image of the character, T represents the temperature hyperparameter, K represents the number of categories, and f is the text line recognition model network. i (x) represents the feature value corresponding to the i-th category out of K categories.

10. The method according to any one of claims 7-9, wherein, Before performing line text recognition on the text line image using the trained text line recognition model, the following steps are also included: The image of the text line to be identified is subjected to image enhancement preprocessing.

11. A text recognition system, wherein, include: The acquisition unit is adapted to acquire the image of the text line to be recognized; The text line recognition unit is adapted to perform line text recognition on the text line image using a trained text line recognition model to obtain the text line recognition result; The evaluation unit is adapted to evaluate the confidence and energy of each character in the text line recognition result; The cropping unit is adapted to crop a character sub-image of a preset width centered on the corresponding character in the text line image when the evaluation unit determines that the confidence of any character in the text line recognition result is less than a preset confidence threshold or the energy of the character is greater than a preset energy threshold. The character recognition unit is suitable for using a trained single-character detection and recognition model to perform single-character detection and recognition for each character sub-image, and outputs the corresponding character recognition results; The recognition result output unit is adapted to combine the text line recognition result and the corresponding character recognition result to obtain and output the character recognition result of the text line image.

12. An electronic device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, wherein, When the processor runs the computer program, it performs the steps of the text recognition model generation method according to any one of claims 1 to 6 or the text recognition method according to any one of claims 7 to 10.

13. A computer-readable storage medium having a computer program stored thereon, wherein, When the computer program is run, it performs the steps of the text recognition model generation method according to any one of claims 1 to 6 or the text recognition method according to any one of claims 7 to 10.

Images