Method for recognizing characters and related products

Abstract

Embodiments of the present application provide a method for recognizing characters and related products. The method comprises: obtaining a single-line text image to be recognized, wherein the single-line text image includes one or more language characters; processing the single-line text image based on a character recognition model in an inference stage, wherein the character recognition model in the training stage and the inference stage adopts different network structures, and the character recognition model in the inference stage is obtained by lightweight processing of the character recognition model trained in the training stage; and determining the recognition results of all language characters in the single-line text image based on the output of the character recognition model in the inference stage. Through the technical solution of the present application, the recognition of multi-language characters can be realized by combining a lightweight character recognition model, and the entire recognition process does not need to explicitly distinguish the target language, which can greatly improve the character recognition efficiency.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of computer technology, and more specifically, to a method for recognizing text, as well as an electronic device and a computer-readable storage medium for performing the aforementioned method. Background Technology

[0002] This section is intended to provide background or context for embodiments of the invention set forth in the claims. The description herein may include concepts that may be explored, but not necessarily concepts that have been previously conceived or explored. Therefore, unless otherwise stated, what is described in this section is not prior art for the purposes of this application's specification and claims, and is not acknowledged as prior art simply by virtue of its inclusion in this section.

[0003] As a smart hardware product equipped with Optical Character Recognition (OCR) functionality, the dictionary pen, in scanning and recognition scenarios, requires the ability to recognize text from multi-frame stitched text images captured by a high-speed camera, as well as high recognition speed, low latency, and high accuracy. However, in actual scanning and recognition scenarios, users scan a wide variety of materials, making the text recognition task quite complex. These include multilingual materials (Chinese, English, Japanese, and Korean), textbooks, test papers, workbooks, children's picture books, newspapers and magazines, and handwritten text, which contain different text formats such as horizontal / vertical and various font styles and background styles, posing a challenge to the generalization ability of the dictionary pen's text recognition function.

[0004] In related technologies, for complex language character recognition tasks (especially multilingual character recognition tasks), the target language corresponding to the image content to be recognized is usually determined first, and then character recognition is performed for each language. This text recognition method not only limits the text recognition scenarios (e.g., it cannot directly recognize characters in images with mixed text in multiple languages), but also directly affects the text recognition results if the target language is incorrectly identified. Furthermore, some related technologies employ more complex character recognition systems to ensure accurate results, significantly increasing the cost of text recognition. Moreover, the overly complex structure of these systems makes them unsuitable for low-computing-power platforms like dictionary pens. Summary of the Invention

[0005] The recognition of scanned text is known to be unsatisfactory, which is a very frustrating process.

[0006] Therefore, there is a great need for an improved solution for text recognition that can achieve efficient and accurate text recognition to meet practical needs.

[0007] In this context, embodiments of the present invention aim to provide a method and related products for recognizing text.

[0008] In a first aspect of the present invention, a method for recognizing text is proposed, comprising: acquiring a single-line text image to be recognized, wherein the single-line text image includes one or more language characters; processing the single-line text image based on a character recognition model in the inference phase, wherein the character recognition model in the training phase and the character recognition model in the inference phase employ different network structures, and the character recognition model in the inference phase is obtained by lightweighting the character recognition model trained in the training phase; and determining the recognition results of all language characters in the single-line text image based on the output of the character recognition model in the inference phase.

[0009] In one embodiment of the present invention, the text recognition model in the training phase is a multi-task learning framework, and includes an image text feature encoder for extracting image text features, a first decoder for predicting the probability of characters in a text sequence, a second decoder to assist the first decoder in learning, and / or a text counting model for predicting the number of characters.

[0010] In another embodiment of the present invention, wherein the first decoder comprises a first connected temporal classification CTC decoder consisting of fully connected layers, and the second decoder comprises a second connected temporal classification CTC decoder as a temporal model, and the text recognition model is trained by the following operations: extracting image text features from training samples based on the image text feature encoder; converting the image text features into text sequence features; predicting the text sequence features using the first CTC decoder and the second CTC decoder respectively, and / or predicting the image text features using the text counting model; determining the loss of the text recognition model based on the prediction results of the first CTC decoder, the second CTC decoder and / or the prediction results of the text counting model; and iteratively optimizing the model parameters of the text recognition model based on the loss of the text recognition model until the text recognition model training converges.

[0011] In another embodiment of the present invention, the prediction result of the first CTC decoder includes a first predicted text sequence character probability distribution, the prediction result of the second CTC decoder includes a second predicted text sequence character probability distribution, and the prediction result of the character counting module includes a predicted character quantity probability distribution. Determining the loss of the character recognition model based on the prediction results of the first CTC decoder, the second CTC decoder, and the prediction result of the character counting model includes: calculating the losses between the first predicted text sequence character probability distribution and the second predicted text sequence character probability distribution and the real text line label sequence, respectively, to obtain a first CTC loss and a second CTC loss; calculating the KL divergence loss between the first predicted text sequence character probability distribution and the second predicted text sequence character probability distribution; calculating the cross-entropy loss between the predicted character quantity probability distribution and the real character quantity; and determining the loss of the character recognition model based on the first CTC loss, the second CTC loss, the KL divergence loss, and the cross-entropy loss.

[0012] In another embodiment of the present invention, determining the loss of the character recognition model based on the first CTC loss, the second CTC loss, the KL divergence loss, and the cross-entropy loss includes: performing a weighted summation of the first CTC loss, the second CTC loss, the KL divergence loss, and the cross-entropy loss, and determining the weighted summation result as the loss of the character recognition model.

[0013] In one embodiment of the present invention, processing the single-line text image based on the text recognition model in the inference stage includes: cropping and quantizing the trained text recognition model to obtain the text recognition model in the inference stage; and deploying the text recognition model in the inference stage offline in an electronic device to perform processing on the single-line text image.

[0014] In another embodiment of the invention, the text recognition model in the inference phase includes the image text feature encoder and the first decoder.

[0015] In another embodiment of the present invention, the image text feature encoder includes a convolutional neural network, and the first decoder includes a first CTC decoder.

[0016] In another embodiment of the present invention, determining the recognition results of all languages ​​in the single-line text image based on the output of the text recognition model in the inference stage includes: extracting the maximum predicted probability category identifier of each character from the probability distribution of the text sequence output by the text recognition model in the inference stage; and obtaining the text corresponding to the maximum predicted probability category identifier of each character based on the predefined mapping relationship between character identifiers and text.

[0017] In a second aspect of the present invention, an electronic device is provided, comprising: a processor; and a memory storing computer instructions for recognizing text, wherein when the computer instructions are executed by the processor, the electronic device causes the electronic device to perform the method described in accordance with the foregoing and the following embodiments.

[0018] In a third aspect of the present invention, a computer-readable storage medium is provided, comprising program instructions for recognizing text, which, when executed by a processor, cause the methods described according to the foregoing and the following plurality of embodiments to be implemented.

[0019] In a fourth aspect of the present invention, an apparatus for recognizing text is provided, comprising: an acquisition unit for acquiring a single-line text image to be recognized, wherein the single-line text image includes one or more language characters; a processing unit for processing the single-line text image based on a character recognition model in the inference stage, wherein the character recognition models in the training stage and the inference stage employ different network structures, and the character recognition model in the inference stage is obtained by lightweight processing of the character recognition model trained in the training stage; and a recognition unit for determining the recognition results of all language characters in the single-line text image based on the output of the character recognition model in the inference stage.

[0020] In one embodiment of the present invention, the text recognition model in the training phase is a multi-task learning framework, and includes an image text feature encoder for extracting image text features, a first decoder for predicting the probability of characters in a text sequence, a second decoder to assist the first decoder in learning, and / or a text counting model for predicting the number of characters.

[0021] In another embodiment of the present invention, wherein the first decoder includes a first connected temporal classification CTC decoder composed of fully connected layers, and the second decoder includes a second connected temporal classification CTC decoder as a temporal model, and the processing unit is further configured to train the text recognition model by: extracting image text features from training samples based on the image text feature encoder; converting the image text features into text sequence features; predicting the text sequence features using the first CTC decoder and the second CTC decoder respectively, and / or predicting the image text features using the text counting model; determining the loss of the text recognition model based on the prediction results of the first CTC decoder, the second CTC decoder and / or the prediction results of the text counting model; and iteratively optimizing the model parameters of the text recognition model based on the loss of the text recognition model until the text recognition model training converges.

[0022] In another embodiment of the present invention, the prediction result of the first CTC decoder includes a first predicted text sequence character probability distribution, the prediction result of the second CTC decoder includes a second predicted text sequence character probability distribution, and the prediction result of the character counting module includes a predicted character quantity probability distribution. The processing unit is specifically configured to: calculate the losses between the first predicted text sequence character probability distribution and the second predicted text sequence character probability distribution and the actual text line label sequence, respectively, to obtain a first CTC loss and a second CTC loss; calculate the KL divergence loss between the first predicted text sequence character probability distribution and the second predicted text sequence character probability distribution; calculate the cross-entropy loss between the predicted character quantity probability distribution and the actual character quantity; and determine the loss of the character recognition model based on the first CTC loss, the second CTC loss, the KL divergence loss, and the cross-entropy loss.

[0023] In another embodiment of the present invention, the processing unit is further specifically configured to: perform a weighted summation of the first CTC loss, the second CTC loss, the KL divergence loss, and the cross-entropy loss, and determine the weighted summation result as the loss of the character recognition model.

[0024] In one embodiment of the present invention, the processing unit is further specifically configured to: perform cropping and quantization processing on the trained character recognition model to obtain the character recognition model in the inference stage; and

[0025] The text recognition model in the inference stage is deployed offline in an electronic device to perform processing on the single-line text image.

[0026] In another embodiment of the invention, the text recognition model in the inference phase includes the image text feature encoder and the first decoder.

[0027] In another embodiment of the present invention, the image text feature encoder includes a convolutional neural network, and the first decoder includes a first CTC decoder.

[0028] In another embodiment of the present invention, the recognition unit is specifically used to: extract the maximum predicted probability category identifier of each character from the probability distribution of the text sequence output by the text recognition model in the inference stage; and obtain the text corresponding to the maximum predicted probability category identifier of each character based on the predefined mapping relationship between character identifiers and text.

[0029] The method and related products for recognizing text according to embodiments of the present invention can recognize single-line text images containing one or more languages ​​based on a lightweight text recognition model. It can be seen that the solution of the present invention can achieve multilingual text recognition by combining a lightweight text recognition model, and the entire recognition process does not require explicit differentiation of the target language, which can significantly improve text recognition efficiency. In some embodiments, the text recognition model is a multi-task learning framework, and the text recognition model can be trained using a first CTC decoder, a second CTC decoder, and a character counting model, so that the trained text recognition model can have high text recognition accuracy and model generalization performance for multiple languages ​​or complex scanning scenarios.

[0030] In other embodiments, the trained text recognition model can be cropped and quantized to achieve lightweight deployment on low-computing platforms, thereby ensuring text recognition accuracy while significantly improving the model's inference speed. Attached Figure Description

[0031] The above and other objects, features, and advantages of exemplary embodiments of the present invention will become readily apparent from the following detailed description taken in conjunction with the accompanying drawings. Several embodiments of the invention are illustrated in the drawings by way of example and not limitation, wherein:

[0032] Figure 1 A block diagram schematically illustrates an exemplary computing system 100 suitable for implementing embodiments of the present invention;

[0033] Figure 2 A schematic flowchart of a method for recognizing characters according to an embodiment of the present invention is shown.

[0034] Figure 3 A schematic flowchart of a method for recognizing text according to another embodiment of the present invention is shown.

[0035] Figure 4 A schematic diagram illustrating the framework of training and inference of a character recognition model according to an embodiment of the present invention is shown.

[0036] Figure 5 A schematic diagram of the structure of a device for recognizing characters according to an embodiment of the present invention is shown; and

[0037] Figure 6 A schematic diagram of the structure of an electronic device according to an embodiment of the present invention is shown. In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0038] The principles and spirit of the invention will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are given merely to enable those skilled in the art to better understand and implement the invention, and are not intended to limit the scope of the invention in any way. Rather, these embodiments are provided to make this disclosure more thorough and complete, and to fully convey the scope of this disclosure to those skilled in the art.

[0039] The principles and spirit of the invention will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are given merely to enable those skilled in the art to better understand and implement the invention, and are not intended to limit the scope of the invention in any way. Rather, these embodiments are provided to make this disclosure more thorough and complete, and to fully convey the scope of this disclosure to those skilled in the art.

[0040] Figure 1 A block diagram of an exemplary computing system 100 suitable for implementing embodiments of the present invention is shown. For example... Figure 1 As shown, the computing system 100 may include: a central processing unit (CPU) 101, random access memory (RAM) 102, read-only memory (ROM) 103, a system bus 104, a hard disk controller 105, a keyboard controller 106, a serial interface controller 107, a parallel interface controller 108, a display controller 109, a hard disk 110, a keyboard 111, a serial external device 112, a parallel external device 113, and a display 114. Among these devices, the CPU 101, RAM 102, ROM 103, hard disk controller 105, keyboard controller 106, serial controller 107, parallel controller 108, and display controller 109 are coupled to the system bus 104. The hard disk 110 is coupled to the hard disk controller 105, the keyboard 111 is coupled to the keyboard controller 106, the serial external device 112 is coupled to the serial interface controller 107, the parallel external device 113 is coupled to the parallel interface controller 108, and the display 114 is coupled to the display controller 109. It should be understood that... Figure 1 The structural block diagrams described are for illustrative purposes only and are not intended to limit the scope of the invention. In some cases, certain devices may be added or removed depending on the specific circumstances.

[0041] Those skilled in the art will recognize that embodiments of the present invention can be implemented as a system, method, or computer program product. Therefore, this disclosure can be specifically implemented as entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, generally referred to herein as a "circuit," "module," "unit," or "system." Furthermore, in some embodiments, the present invention can also be implemented as a computer program product contained in one or more computer-readable media, which includes computer-readable program code.

[0042] Any combination of one or more computer-readable media may be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example,, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (not exhaustive) of a computer-readable storage medium may include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device.

[0043] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0044] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0045] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0046] Embodiments of the present invention will now be described with reference to flowchart illustrations and block diagrams of apparatus (or systems) according to embodiments of the invention. It should be understood that each block of the flowchart and / or block diagram, and combinations of blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine that, when executed by the computer or other programmable data processing apparatus, creates means for implementing the functions / operations specified in the blocks of the flowchart and / or block diagram.

[0047] These computer program instructions may also be stored in a computer-readable medium that enables a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable medium produce a product comprising an instruction apparatus that implements the functions / operations specified in the boxes of a flowchart and / or block diagram.

[0048] Computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, such that the instructions that execute on the computer or other programmable apparatus can provide a process for implementing the functions / operations specified in the boxes of a flowchart and / or block diagram.

[0049] According to embodiments of the present invention, a method for recognizing text and related products are proposed. Furthermore, the number of any elements in the accompanying drawings is for illustrative purposes only and not for limitation, and any naming is for distinguishing purposes only and has no limiting meaning.

[0050] The principles and spirit of the present invention will be explained in detail below with reference to several representative embodiments. Invention Overview

[0051] The inventors have discovered that existing methods for recognizing scanned text are not ideal. Specifically, in practical applications, the diverse nature of the scanned materials makes the generalization ability of text recognition models challenging. Specifically, in related technologies, for multilingual text recognition tasks, the target language corresponding to the content of the image to be recognized is usually determined first, and then the text information contained in the image is recognized according to the target language's recognition model. That is, related technologies generally employ a habitual approach of distinguishing between languages. Using the above-mentioned text recognition methods, it is impossible to directly recognize text in images with mixed text content in multiple languages. Furthermore, if the target language is incorrectly identified, it will directly affect the text recognition results. During implementation, there is a risk of missed or incorrect text recognition, resulting in low text recognition accuracy.

[0052] Furthermore, some technical solutions employ complex recognition models (such as recurrent neural networks, RNNs) to improve text recognition accuracy and increase model capacity and computational load, resulting in slow inference speeds on low-computing-power platforms like dictionary pens. Other related technologies, aimed at improving the recognition accuracy of multilingual target text, utilize multiple target language text recognition models simultaneously, leading to overly complex overall text recognition system structures that are difficult to deploy on edge devices such as dictionary pens.

[0053] Based on this, the inventors discovered through research that text recognition models with different structures in the inference and training phases can be used to achieve text recognition of single-line text images. At the same time, without explicitly distinguishing languages ​​during the text recognition process, the accuracy and speed of text recognition can be effectively improved.

[0054] After introducing the basic principles of the present invention, various non-limiting embodiments of the present invention will be described in detail below. Exemplary methods

[0055] The following is for reference. Figure 2 This section describes a method for recognizing text according to an exemplary embodiment of the present invention. It should be noted that the embodiments of the present invention can be applied to any applicable scenario.

[0056] Figure 2 A method 200 for recognizing characters according to an embodiment of the present invention is illustrated schematically.

[0057] like Figure 2As shown, in step S201, a single-line text image to be recognized can be acquired. This single-line text image may include text in one language, or it may include text in multiple languages ​​(e.g., text in a mixed language). In some embodiments, a single-line text image or a multi-line text image can be acquired from the material to be scanned (e.g., multilingual material, newspapers, handwritten text, etc.). The acquired image to be recognized is then processed to obtain the aforementioned single-line text image.

[0058] Preferably, if the acquired image to be recognized is deformed, the layout of graphics and text in the acquired image to be recognized can be further corrected to finally obtain a single-line text image. For example, a corrected single-line text image can be obtained through an image correction module such as the Thin Plate Function Model (TPS).

[0059] It should be noted that the detailed description of the specific acquisition process of single-line text images here is merely illustrative, and the solution of this invention is not limited thereto. For example, the text to be scanned can be selected from various different layouts, font styles, and background styles. The image scanning method for the text to be scanned can be single-line scanning or multi-line scanning, etc. The obtained single-line text image can be adjusted for image correction according to actual processing needs.

[0060] Next, in step S202, the aforementioned single-line text image can be processed based on the character recognition model from the inference stage. This character recognition model employs different network structures in the training and inference stages, and the inference stage model can be a lightweight version of the trained character recognition model. This character recognition model does not target a specific language in the single-line text image, but rather performs non-explicit differentiation across all languages. Furthermore, the inference stage model is a lightweight version of the training stage model, breaking away from the conventional approach of having the same structure in both training and inference stages, thus reducing the complexity of the character recognition model during the character recognition process.

[0061] Finally, in step S203, the recognition results of all languages ​​in the single-line text image can be determined based on the output of the text recognition model in the inference stage. Thus, a lightweight text recognition model can be used to recognize multiple languages, and the entire recognition process does not require explicit differentiation of the target language, significantly improving text recognition efficiency and accuracy.

[0062] Figure 3 A method 300 for recognizing characters according to another embodiment of the present invention is illustrated schematically. It will be understood that method 300 is a... Figure 2 Further limitations or extensions of Chinese method 200. Therefore, the preceding text, combined with... Figure 2The relevant detailed descriptions also apply to the following text.

[0063] like Figure 3 As shown, in step S301, a single-line text image to be recognized can be acquired. As mentioned above, the material to be scanned can be scanned in single-line or multi-line fashion and processed to obtain a single-line text image. The material to be scanned may include text with various layouts, font styles, and background styles.

[0064] Next, in step S302, the trained text recognition model can be pruned and quantized to obtain the text recognition model for the inference stage. In some embodiments, the aforementioned text recognition model for the training stage is a multi-task learning framework and may include an image text feature encoder for extracting image text features, a first decoder for predicting the probability of characters in a text sequence, a second decoder to assist the first decoder in learning, and / or a character counting model for predicting the number of characters. For example, the text recognition model for the training stage may include a first decoder and a second decoder. Alternatively, the text recognition model for the training stage may include a first decoder, a second decoder, and a character counting model. It is understood that the specific network structure of the text recognition model for the training stage can be adjusted according to the actual task requirements.

[0065] In some embodiments, the first decoder may include a first connected temporal classification CTC decoder (hereinafter referred to as the first CTC decoder), and the second decoder may include a second connected temporal classification CTC decoder (hereinafter referred to as the second CTC decoder). The first CTC decoder may consist only of fully connected layers and has the advantages of being lightweight and having fast text decoding speed, while the second CTC decoder may consist of a temporal model (such as a recurrent neural network (RNN) or a bidirectional short-time memory network). During the training of the text recognition model, image text features can be extracted from training samples based on the aforementioned image text feature encoder (such as a convolutional neural network (CNN)). Then, the image text features are converted into text sequence features, and the first and second CTC decoders are used to predict the text sequence features, and / or a word counting model is used to predict the image text features. Then, the loss of the text recognition model is determined based on the prediction results of the first and second CTC decoders and / or the prediction results of the word counting model, and the model parameters of the text recognition model are iteratively optimized based on the loss of the text recognition model until the text recognition model training converges. This achieves the training of the text recognition model.

[0066] In some embodiments, the prediction result of the aforementioned first CTC decoder may include a first predicted text sequence character probability distribution, the prediction result of the second CTC decoder may include a second predicted text sequence character probability distribution, and the prediction result of the aforementioned character counting module may include a predicted character count probability distribution. Therefore, the losses between the first and second predicted text sequence character probability distributions and the actual text line label sequence can be calculated to obtain the first CTC loss and the second CTC loss. Simultaneously, the KL divergence loss between the first and second predicted text sequence character probability distributions and the cross-entropy loss between the predicted character count probability distribution and the actual character count can be calculated. Then, the loss of the aforementioned character recognition model can be determined based on the first CTC loss, the second CTC loss, the KL divergence loss, and the cross-entropy loss.

[0067] It is understandable that the specific calculation process of the loss of the character recognition model can be adaptively adjusted according to the specific structure of the character recognition model. For example, if the character recognition model includes a character counting model, then a cross-entropy loss is added accordingly when calculating the loss. If the character recognition model does not include a character counting model, then only the first CTC loss, the second CTC loss, and the KL divergence loss can be used to determine the loss of the character recognition model. Specifically, in some embodiments, when it is necessary to use the first CTC loss, the second CTC loss, the KL divergence loss, and the cross-entropy loss to determine the loss of the character recognition model, the aforementioned first CTC loss, second CTC loss, KL divergence loss, and cross-entropy loss can be weighted and summed, and the weighted summation result can be determined as the loss of the character recognition model.

[0068] After training the character recognition model, it can be pruned and quantized to obtain the character recognition model for the inference stage. In some embodiments, the character recognition model for the inference stage may include an image text feature encoder (e.g., a convolutional neural network) and a first decoder (e.g., a first CTC decoder composed of fully connected layers). The first CTC decoder's parameters are independently optimized during the training phase, so the accuracy of the pruned character recognition model using the fully connected CTC decoder is not affected, and it does not rely on time-consuming temporal models, significantly improving inference speed. Furthermore, the pruned model is quantized, and the entire recognition model uses low-precision (e.g., int8 data type) to represent the model parameters. Compared to high-precision (e.g., float32 data type) representation, this significantly reduces computational and memory overhead, ensuring that inference speed is greatly improved while maintaining almost no loss in character recognition accuracy.

[0069] After obtaining the text recognition model in the inference stage, in step S303, the aforementioned text recognition model in the inference stage can be deployed offline in an electronic device to perform the processing of the aforementioned single-line text image.

[0070] The following combination Figure 4 The training and inference processes of the character recognition model are explained in detail. For example... Figure 4 As shown, the character recognition model employs a multi-task learning framework. The model training phase mainly involves training sample generation, the multi-task learning framework, and a multilingual character recognition model. The inference phase mainly involves character recognition model pruning, quantization deployment, and model inference. It is understandable that... Figure 4 In the training phase, the target recognition images are used as training data, while in the inference phase, the target recognition images are used as the actual images to be recognized.

[0071] First, during the training phase, training data can be obtained in various ways. For example, user-reported log data images from edge devices (such as dictionary pens) can be collected, and real training samples (including images and text line label sequence pairs) can be obtained through manual annotation. Alternatively, training data can be synthesized using rendering tools. Specifically, to synthesize data as realistically as possible, edge devices such as dictionary pens can be used to scan various blank materials as background images for synthesized text images. Then, multilingual text corpora and fonts are acquired, and data synthesis is performed using rendering tools. The input includes text corpora, fonts, and background images. By setting attributes such as font size, color, character spacing, stroke thickness, and horizontal / vertical layout, and through rendering, the text and background images are superimposed to obtain synthesized training samples (including text images and text line label sequence pairs).

[0072] Furthermore, image enhancement transformations such as motion blur, perspective distortion, text transparency, and image compression can be applied to the generated training samples to increase sample diversity and realism. Using these methods, and considering the characteristics of multilingual data, training sample pairs in multiple languages, including Chinese, English, Japanese, Korean, and Pinyin, can be generated, providing rich and diverse large-scale training data for model training.

[0073] Next, the model training process mainly involves the multi-task learning process of the text recognition model. For example... Figure 4As shown, the text recognition model in this embodiment consists of a CNN backbone network (using a convolutional neural network as the image text feature encoder), a dual-channel CTC decoder structure, and a text counting auxiliary branch, forming a multi-task learning framework. The dual-channel CTC decoder structure includes a lightweight fully connected CTC decoder (i.e., the first CTC decoder mentioned earlier, which can consist of only fully connected layers) and a temporal model CTC decoder (i.e., the second CTC decoder mentioned earlier, where the temporal model can use, but is not limited to, RNN / BiLSTM, etc.).

[0074] During training, a CNN, serving as the backbone network, can be used to extract image text features from the training data. Assuming the image text features extracted by the backbone network are represented as N*C*H*W, they can be transformed into a sequence feature representation N*Tseq*Cseq (where Tseq = W, Cseq = H*C) using functions such as reshape. This sequence feature is then passed through a fully connected CTC decoder and a temporal model CTC decoder to obtain the predicted character class probability distribution N*Tseq*Cn, where Tseq is the maximum length of the text sequence, and Cn is the number of predefined dictionary character classes. Then, the CTC loss of the fully connected CTC decoder and the temporal model CTC decoder, respectively, predicting the character probability distribution of the text sequence and the CTC loss of the actual text line label sequence, is calculated to iteratively optimize the text recognition model parameters.

[0075] As can be seen, the aforementioned fully connected CTC decoder and temporal model CTC decoder share the image-text features extracted by the backbone network. The temporal model CTC decoder enhances the model's ability to model the contextual relationships of image-text sequences, updating the backbone network parameters through gradient backpropagation to assist the fully connected CTC decoder in learning. Since the temporal model CTC decoder has a stronger learning ability than the fully connected CTC decoder, its predictions are used as one of the learning objectives of the fully connected CTC decoder. By calculating the KL divergence loss between the probability distributions of the predicted characters from the two decoders, the capability of the fully connected CTC decoder is improved. In this way, the fully connected CTC decoder of the text recognition model not only relies on visual features for text classification but can also learn the contextual relationships of text sequences, which is beneficial for improving the recognition ability of similar-looking characters in multiple languages ​​and reducing misidentification.

[0076] In addition, such as Figure 4As shown, for scenarios involving long text scanning and recognition in dictionary pen applications, an additional character counting auxiliary branch can be added during the character recognition model training phase. This branch predicts the number of characters in the image as one of the learning objectives, reducing missed recognitions. In some embodiments, the character counting auxiliary branch has a structure similar to the image classification model. It can use global average pooling to aggregate the image text features extracted by the backbone network, and then use fully connected layers and activation functions (e.g., softmax) to predict the probability distribution of the number of characters (e.g., including the number of predicted categories and the maximum length of the text sequence). In some embodiments, the character counting auxiliary branch can use a cross-entropy loss function to calculate the cross-entropy loss between the branch's predicted probability distribution of the number of characters in the image and the actual number of characters in the corresponding text line label sequence. Based on the cross-entropy loss, dual-path CTC decoding loss, and KL divergence loss of the dual-path CTC decoding branch, multiplied by the weights of different losses and summed, this becomes the loss of the entire character recognition model. The model parameters of the character recognition model are iteratively optimized using the above losses until the character recognition model training converges.

[0077] The character recognition model trained based on the aforementioned multi-task learning framework can perform multilingual character recognition without explicitly distinguishing the target language, using only a single model. It supports Chinese, English, Japanese, and Korean character recognition simultaneously. Through learning from large-scale labeled training samples, it effectively improves character recognition accuracy and model generalization performance, achieving accurate character recognition results in complex scenarios such as dictionary pen scanning recognition. It also handles the recognition of common and rare characters, supports handwritten characters, Pinyin, and artistic characters, and supports horizontal / vertical text recognition.

[0078] During the model inference phase, the pre-trained character recognition model within the multi-task learning framework can be pruned and quantized for offline deployment on low-computing-power electronic devices such as dictionary pens. Specifically, by pruning the temporal model CTC decoder and character counting branch that assists in the character recognition model's learning, only the backbone network and a lightweight fully connected CTC decoder are retained for multilingual character recognition. Since the fully connected CTC decoder's parameters are independently optimized during training, the pruned character recognition model's accuracy using the fully connected CTC decoder remains unaffected, while also significantly improving inference speed by avoiding reliance on the time-consuming temporal model.

[0079] After deploying the text recognition model, return Figure 3In step S304, the maximum predicted probability category identifier for each character can be extracted from the probability distribution of the text sequence output by the text recognition model in the aforementioned inference stage. Finally, in step S305, the text corresponding to the maximum predicted probability category identifier for each character can be obtained based on the predefined mapping relationship between character identifiers and text. Specifically, in some embodiments, functions (such as the argmax function) can be used to extract the maximum predicted probability category ID for each character from the probability distribution of the sequence characters predicted by the trained text recognition model, and redundancy removal processing can be performed. Then, the text recognition result of the target recognition image can be obtained by transcribing according to the mapping relationship of character IDs in the predefined character table.

[0080] As can be seen, the solution of this invention can achieve multilingual character recognition using a single character recognition model without explicitly distinguishing the target language. It supports Chinese, English, Japanese, and Korean character recognition, taking into account both common and rare characters, as well as handwritten characters, Pinyin, artistic characters, and horizontal / vertical text recognition. It is also more effective at recognizing long and complex texts. The entire character recognition model training process is concise and efficient. No post-processing of the recognition results is required during the inference stage, greatly simplifying the character recognition system. It can be deployed offline on low-computing-power platforms on the dictionary pen. In addition, the multilingual character recognition model in this invention benefits from a multi-task learning training framework. During the model training stage, the character counting branch and the temporal model CTC decoder can be used to assist the lightweight fully connected CTC decoder in learning. By learning from large-scale labeled data, the recognition accuracy is effectively improved, and accurate character recognition results can be obtained in complex scenarios such as dictionary pen scanning recognition. During the inference stage, it does not rely on time-consuming temporal model structures such as RNN / LSTM. The entire recognition model can use low-precision data types to quantize the model parameters, which significantly reduces the computational load and memory overhead compared to high-precision data type representations. Furthermore, the text recognition model in the inference stage has been pruned and quantized for deployment, significantly improving inference speed with almost no loss of accuracy. This results in faster text recognition speeds on edge devices such as dictionary pens, meeting real-time scanning and recognition requirements. It also provides smoother performance in single-line, multi-line, and paragraph scanning, recognition, translation, and word lookup scenarios, enhancing the user experience. Exemplary device

[0081] After introducing the method of exemplary embodiments of the present invention, the following references are made. Figure 5 and Figure 6 The present invention describes related products for recognizing text according to exemplary embodiments of the present invention.

[0082] Figure 5 A schematic diagram illustrating a possible structure of a device 500 for recognizing characters according to an embodiment of the present invention is shown. It should be noted that... Figure 5 The middle device 500 can perform Figure 2 and Figure 3 The method described above. Therefore, in conjunction with the preceding text... Figure 2 and Figure 3 The relevant detailed descriptions also apply below. Additionally, Figure 5 The training and inference processes of the character recognition model can be found in [reference]. Figure 4 Therefore, the preceding text combined Figure 4 The relevant detailed descriptions also apply below.

[0083] like Figure 5 As shown, the device 500 may include an acquisition unit 501, a processing unit 502, and a recognition unit 503. The acquisition unit 501 acquires a single-line text image to be recognized. The single-line text image includes one or more languages. The processing unit 502 processes the single-line text image based on a character recognition model in the inference stage, wherein the character recognition models in the training and inference stages employ different network structures, and the character recognition model in the inference stage is obtained by lightweighting the character recognition model trained in the training stage. In some embodiments, the processing unit is further specifically used to: crop and quantize the trained character recognition model to obtain the character recognition model in the inference stage, and deploy the character recognition model in the inference stage offline in an electronic device to perform processing on the single-line text image. The recognition unit 503 is specifically used to: extract the maximum predicted probability category identifier of each character from the probability distribution of the text sequence output by the character recognition model in the inference stage, and obtain the character corresponding to the maximum predicted probability category identifier of each character based on a predefined mapping relationship between character identifiers and characters. The various processing procedures involved in the processing unit and the recognition unit can be referred to in conjunction with the preceding text. Figures 2-4 The relevant descriptions in the text will not be repeated here.

[0084] It should be noted that in practical applications, the various modules in device 500 can be further split, deleted, or combined according to the actual application requirements. There are no restrictions on the specific architecture that device 500 can contain.

[0085] Figure 6 A schematic block diagram of an electronic device 600 according to an embodiment of the present invention is shown. Figure 6 As shown, the electronic device 600 may include a processor 601 and a memory 602. The memory 602 stores computer instructions for recognizing text. When these computer instructions are executed by the processor 601, the electronic device 600 performs actions according to the preceding description. Figures 2 to 3The described method. For example, in some embodiments, electronic device 300 can acquire a single-line text image, train a text recognition model, perform lightweight processing on the trained text recognition model, and use the lightweight processed text recognition model to recognize all languages ​​of the single-line text image, etc. Based on this, electronic device 600 can intelligently and dynamically customize a preparation plan for the user, effectively improving the accuracy and speed of text recognition.

[0086] In some implementation scenarios, electronic device 600 may include dictionary pen, scanning translation pen, and other electronic devices with text recognition requirements (especially multilingual text recognition requirements). The present invention does not limit the structural design that electronic device 600 may have.

[0087] It should be noted that although several devices or sub-devices of the device have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of the invention, the features and functions of two or more devices described above can be embodied in one device. Conversely, the features and functions of one device described above can be further divided and embodied by multiple devices.

[0088] The use of the verbs "including" and "contains" and their inflections in the application documents does not preclude the existence of elements or steps other than those described in the application documents. The article "a" or "one" preceding an element does not preclude the existence of multiple such elements.

[0089] While the spirit and principles of the invention have been described with reference to several specific embodiments, it should be understood that the invention is not limited to the disclosed specific embodiments, and the division of aspects does not imply that features in these aspects cannot be combined for benefit; such division is merely for ease of description. The invention is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims. The scope of the appended claims is to be interpreted in the broadest sense, thereby encompassing all such modifications and equivalent structures and functions.

Claims

1. A method for recognizing characters, characterized in that, include: Acquire a single-line text image to be identified, wherein the single-line text image includes one or more languages; and The single-line text image is processed based on the text recognition model in the inference stage. The text recognition models in the training stage and the inference stage adopt different network structures. The text recognition model in the inference stage is obtained by lightweight processing of the text recognition model trained in the training stage. The text recognition model in the inference stage does not explicitly distinguish between all languages ​​in the single-line text image. The text recognition model in the training stage is a multi-task learning framework and includes an image text feature encoder for extracting image text features, a first decoder for predicting the probability of characters in the text sequence, a second decoder to assist the first decoder in learning, and / or a character counting model for predicting the number of characters. The text recognition model in the inference stage includes the image text feature encoder and the first decoder. The first decoder includes a first CTC decoder composed of fully connected layers, and the parameters of the first CTC decoder are independently optimized in the training stage. as well as Based on the output of the text recognition model in the inference stage, the recognition results of all languages ​​in the single-line text image are determined.

2. The method according to claim 1, characterized in that, The second decoder includes a second CTC decoder as a temporal model, and the text recognition model is trained via the following operations: Based on the image text feature encoder, image text features are extracted from the training samples; Convert the image text features into text sequence features; The text sequence features are predicted using the first CTC decoder and the second CTC decoder, respectively, and / or the image text features are predicted using the text counting model; The loss of the character recognition model is determined based on the prediction results of the first CTC decoder, the second CTC decoder, and / or the prediction results of the character counting model. as well as The model parameters of the character recognition model are iteratively optimized based on the loss of the character recognition model until the training of the character recognition model converges.

3. The method according to claim 2, characterized in that, in, The prediction result of the first CTC decoder includes a first predicted text sequence character probability distribution; the prediction result of the second CTC decoder includes a second predicted text sequence character probability distribution; and the prediction result of the character counting model includes a predicted character count probability distribution. The loss of the character recognition model is determined based on the prediction results of the first CTC decoder, the second CTC decoder, and the character counting model, including: Calculate the loss between the first predicted text sequence character probability distribution, the second predicted text sequence character probability distribution and the real text line label sequence, respectively, to obtain the first CTC loss and the second CTC loss; Calculate the KL divergence loss between the character probability distributions of the first predicted text sequence and the character probability distributions of the second predicted text sequence; Calculate the cross-entropy loss between the predicted character count probability distribution and the actual character count; and The loss of the character recognition model is determined based on the first CTC loss, the second CTC loss, the KL divergence loss, and the cross-entropy loss.

4. The method according to claim 3, characterized in that, The loss of the character recognition model is determined based on the first CTC loss, the second CTC loss, the KL divergence loss, and the cross-entropy loss, including: The first CTC loss, the second CTC loss, the KL divergence loss, and the cross-entropy loss are weighted and summed, and the weighted summation result is determined as the loss of the character recognition model.

5. The method according to any one of claims 1 to 4, characterized in that, The processing of the single-line text image based on the text recognition model in the inference stage includes: The trained character recognition model is cropped and quantized to obtain the character recognition model for the inference stage; and The text recognition model in the inference stage is deployed offline in an electronic device to perform processing on the single-line text image.

6. The method according to claim 5, characterized in that, Based on the output of the text recognition model in the inference stage, the recognition results of all languages ​​in the single-line text image are determined as follows: Extract the maximum predicted probability category identifier for each character from the probability distribution of the text sequence output by the text recognition model during the inference phase; and Based on the predefined mapping relationship between character identifiers and text, obtain the text corresponding to the highest predicted probability category identifier of each character.

7. The method according to claim 5, characterized in that, The image text feature encoder includes a convolutional neural network.

8. An electronic device, characterized in that, include: processor; as well as A memory that stores computer instructions for recognizing text, which, when executed by the processor, cause the electronic device to perform the method according to any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, It includes program instructions for recognizing text, which, when executed by a processor, cause the method according to any one of claims 1-7 to be implemented.

10. A device for recognizing characters, characterized in that, include: An acquisition unit is used to acquire a single-line text image to be recognized, wherein the single-line text image includes one or more languages. as well as The processing unit is used to process the single-line text image based on the character recognition model in the inference stage. The character recognition models in the training stage and the inference stage adopt different network structures, and the character recognition model in the inference stage is obtained by lightweight processing of the character recognition model trained in the training stage. The character recognition model in the inference stage does not explicitly distinguish between all languages ​​in the single-line text image. The character recognition model in the training stage is a multi-task learning framework and includes an image character feature encoder for extracting image text features, a first decoder for predicting the probability of characters in the text sequence, a second decoder to assist the first decoder in learning, and / or a character counting model for predicting the number of characters. The character recognition model in the inference stage includes the image character feature encoder and the first decoder. The first decoder includes a first CTC decoder composed of fully connected layers, and the parameters of the first CTC decoder are independently optimized in the training stage. as well as The recognition unit is used to determine the recognition results of all languages ​​in the single-line text image based on the output of the text recognition model in the inference stage.

11. The apparatus according to claim 10, characterized in that, The second decoder includes a second connection-temporal classification (CTC) decoder as a temporal model, and the processing unit is further configured to train the text recognition model through the following operations: Based on the image text feature encoder, image text features are extracted from the training samples; Convert the image text features into text sequence features; The text sequence features are predicted using the first CTC decoder and the second CTC decoder, respectively, and / or the image text features are predicted using the text counting model; The loss of the character recognition model is determined based on the prediction results of the first CTC decoder, the second CTC decoder, and / or the prediction results of the character counting model. as well as The model parameters of the character recognition model are iteratively optimized based on the loss of the character recognition model until the training of the character recognition model converges.

12. The apparatus according to claim 11, characterized in that, in, The prediction result of the first CTC decoder includes a first predicted text sequence character probability distribution; the prediction result of the second CTC decoder includes a second predicted text sequence character probability distribution; the prediction result of the character counting model includes a predicted character count probability distribution; and the processing unit is specifically used for: Calculate the loss between the first predicted text sequence character probability distribution, the second predicted text sequence character probability distribution and the real text line label sequence, respectively, to obtain the first CTC loss and the second CTC loss; Calculate the KL divergence loss between the character probability distributions of the first predicted text sequence and the character probability distributions of the second predicted text sequence; Calculate the cross-entropy loss between the predicted character count probability distribution and the actual character count; as well as The loss of the character recognition model is determined based on the first CTC loss, the second CTC loss, the KL divergence loss, and the cross-entropy loss.

13. The apparatus according to claim 12, characterized in that, The processing unit is also specifically used for: The first CTC loss, the second CTC loss, the KL divergence loss, and the cross-entropy loss are weighted and summed, and the weighted summation result is determined as the loss of the character recognition model.

14. The apparatus according to any one of claims 11 to 13, characterized in that, The processing unit is also specifically used for: The trained character recognition model is cropped and quantized to obtain the character recognition model for the inference stage; and The text recognition model in the inference stage is deployed offline in an electronic device to perform processing on the single-line text image.

15. The apparatus according to claim 14, characterized in that, The image text feature encoder includes a convolutional neural network.

16. The apparatus according to claim 14, characterized in that, The identification unit is specifically used for: Extract the maximum predicted probability category identifier for each character from the probability distribution of the text sequence output by the text recognition model during the inference phase; and Based on the predefined mapping relationship between character identifiers and text, obtain the text corresponding to the highest predicted probability category identifier of each character.

Images