Training methods for character recognition models, character recognition methods and related equipment

Abstract

This application discloses a training method for a character recognition model, a character recognition method, and related equipment. The training method includes: acquiring a training dataset, which includes sample images and corresponding ground truth labels for the sample images, wherein the ground truth labels are formed by concatenating the positional information of text regions in the sample images and the actual text in the text regions; performing masking processing on the actual text in the ground truth labels to obtain a first mask label; encoding the sample images through an encoding module in the character recognition model to obtain a first image feature vector of the sample images, and performing character recognition on the first image feature vector based on the first mask label through a decoding module in the character recognition model to obtain a first character recognition result corresponding to the sample images; and optimizing the character recognition model based on the ground truth labels corresponding to the sample images and the first character recognition result.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a training method for a character recognition model, a character recognition method, and related equipment. Background Technology

[0002] OCR (Optical Character Recognition) technology refers to the use of optical and computer technologies to detect and identify text in images, and is a branch of computer vision research. The concept was first proposed in 1929 by the German scientist Tausheck. After nearly a century of development, OCR has achieved significant results in various fields. Text recognition has many applications, including document recognition, road sign recognition, license plate recognition, and industrial number recognition. It is currently widely used in industries such as healthcare and education. Integrating theoretical knowledge from digital image processing, computer graphics, and artificial intelligence, it is increasingly becoming a focus of attention in the field of artificial intelligence. While traditional OCR technology can achieve high accuracy in specific scenarios involving printed characters, its accuracy is lower in complex scenes due to issues such as lighting, shape, and blur. With the development of artificial intelligence, deep learning-based methods have achieved better results. These methods do not require feature processing and can recognize text in complex scenes, outperforming traditional text recognition methods.

[0003] Currently, traditional text recognition algorithms perform well in simple scenarios, but their accuracy suffers in complex scenarios due to factors such as lighting, shape, and blur. While deep learning-based text recognition algorithms, developed with the advancement of artificial intelligence, are unaffected by lighting conditions and offer high accuracy, they involve a two-stage process: first, deep learning is used to identify text blocks in the image, and then text recognition technology is used to verify those blocks. This results in low efficiency and high complexity. Summary of the Invention

[0004] The purpose of this application is to provide a method for training a character recognition model, a character recognition method, and related equipment, which can realize character recognition in an end-to-end manner while ensuring recognition accuracy, thereby simplifying the recognition process and improving the recognition rate.

[0005] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0006] In a first aspect, embodiments of this application provide a method for training a character recognition model, comprising:

[0007] Obtain a training dataset, which includes sample images and corresponding real labels for the sample images. The real labels are formed by concatenating the positional information of the text regions in the sample images with the real text in the text regions.

[0008] The real text in the real label is masked to obtain the first masked label;

[0009] The sample image is encoded by the encoding module in the character recognition model to obtain the first image feature vector of the sample image. The character recognition model is then decoded by the decoding module based on the first mask label to perform character recognition on the first image feature vector to obtain the first character recognition result corresponding to the sample image.

[0010] Based on the real labels corresponding to the sample images and the first character recognition result, the character recognition model is optimized.

[0011] The text recognition model training method provided in this application embodiment concatenates the location information of text regions in a sample image with the real text within those regions to form a real label corresponding to the sample image. The text recognition model is then trained using the sample image and its corresponding real label. The real label provides a supervisory signal for the training process, enabling the text recognition model to fully learn and understand the image features of the text regions in the sample image and the features of the real text within those regions from the real label, thus simultaneously possessing the ability to detect and recognize text. Specifically, a text generation task is introduced. A first mask label is obtained by masking the real text in the real label. The sample image is then encoded by the encoding module in the text recognition model, and the text recognition model's decoding module performs text recognition based on the encoded first image feature vector using the first mask label. The encoding module obtains the first character recognition result corresponding to the sample image, and optimizes the character recognition model based on the real label corresponding to the sample image and the first character recognition result. This enables the encoding module to accurately encode the relevant image features of the input image. Furthermore, since the unmasked content in the first mask label is the context related to the masked real text, the decoding module can not only learn the image features of the text region's location from the position information in the first mask label to master character detection knowledge, but also predict the masked real text by learning and understanding the semantic context of the masked real text. In other words, it fully learns semantic information and the knowledge of text generation based on semantic information during the decoding process, thus enabling accurate character recognition based on the detected text region. Therefore, the decoder simultaneously possesses the ability to detect and recognize characters, and both detection and recognition accuracy are high.

[0012] Secondly, embodiments of this application provide a character recognition method, including:

[0013] Acquire the image to be recognized;

[0014] The image to be recognized is encoded by the encoding module in the character recognition model to obtain the image feature vector of the image to be recognized, and the character recognition is performed on the image feature vector by the decoding module in the character recognition model to obtain the character recognition result corresponding to the image to be recognized. The character recognition model is trained based on the training method of the character recognition model described in the first aspect.

[0015] The text recognition method provided in this application has the ability to accurately encode the relevant image features of the input image by means of the encoder in the trained text recognition model, and the ability to both detect and recognize text by means of the decoder in the text recognition model. By using the encoding module in the trained text recognition model to encode the image to be recognized, and then using the decoding module in the text recognition model to recognize the text by means of the encoded image feature vector, end-to-end text detection and recognition can be achieved while ensuring the accuracy of text recognition, without the need for a two-stage algorithm that first detects text and then uses the relevant text recognition algorithm to recognize text. This simplifies the text recognition process and improves recognition efficiency.

[0016] Thirdly, embodiments of this application provide a training apparatus for a character recognition model, comprising:

[0017] The first acquisition unit is used to acquire a training dataset, which includes sample images and corresponding real labels for the sample images. The real labels are formed by concatenating the position information of the text regions in the sample images and the real text in the text regions.

[0018] The first masking unit is used to mask the real text in the real label to obtain the first masked label.

[0019] The first recognition unit is used to encode the sample image through the encoding module in the character recognition model to obtain the first image feature vector of the sample image, and to perform character recognition on the first image feature vector based on the first mask label through the decoding module in the character recognition model to obtain the first character recognition result corresponding to the sample image.

[0020] The first optimization unit is used to optimize the text recognition model based on the real label corresponding to the sample image and the first text recognition result.

[0021] Fourthly, embodiments of this application provide a character recognition device, including:

[0022] The second acquisition unit is used to acquire the image to be recognized;

[0023] The second recognition unit is used to encode the image to be recognized through the encoding module in the character recognition model to obtain the image feature vector of the image to be recognized, and to perform character recognition on the image feature vector through the decoding module in the character recognition model to obtain the character recognition result corresponding to the image to be recognized. The character recognition model is trained based on the training method of the character recognition model described in the first aspect.

[0024] Fifthly, embodiments of this application provide an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the method as described in the first aspect; or, the processor is configured to execute the instructions to implement the method as described in the second aspect.

[0025] In a sixth aspect, embodiments of this application provide a computer-readable storage medium that, when the instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to perform the method described in the first aspect; or, when the instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to perform the method described in the second aspect. Attached Figure Description

[0026] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0027] Figure 1 A flowchart illustrating a method for training a character recognition model, provided as an embodiment of this application;

[0028] Figure 2 A schematic diagram illustrating a method for constructing a real label according to an embodiment of this application;

[0029] Figure 3 A flowchart illustrating a method for training a character recognition model, provided as another embodiment of this application;

[0030] Figure 4 A schematic diagram of a preset dictionary provided for one embodiment of this application;

[0031] Figure 5 A schematic diagram of a first mask label acquisition method provided for one embodiment of this application;

[0032] Figure 6 A flowchart illustrating a method for training a character recognition model, provided as another embodiment of this application.

[0033] Figure 7A flowchart illustrating a character recognition method provided in one embodiment of this application;

[0034] Figure 8 A flowchart illustrating a character recognition method provided for another embodiment of this application;

[0035] Figure 9 A schematic diagram of the structure of a training device for a character recognition model provided in one embodiment of this application;

[0036] Figure 10 A schematic diagram of the structure of a character recognition device provided in one embodiment of this application;

[0037] Figure 11 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this application. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0039] The terms "first," "second," etc., used in this specification and claims are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in sequences other than those illustrated or described herein. Furthermore, in this specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0040] Explanation of some concepts:

[0041] OCR (optical character recognition): also known as character recognition, refers to the process by which electronic devices (such as scanners or digital cameras) examine characters printed on paper and then use character recognition methods to translate the shapes into computer text; that is, scanning text materials and then analyzing and processing image files to obtain text and layout information.

[0042] Generative language models are automated models that use computers to generate language text under specific interactive objectives. Their main purpose is to automatically construct high-quality language text that humans can understand.

[0043] As described in the background section, traditional text recognition algorithms perform well in simple scenes, but their accuracy suffers in complex scenes due to factors such as lighting, shape, and blur. With the development of artificial intelligence, deep learning-based text recognition algorithms, while unaffected by lighting conditions and offering high accuracy, involve a two-stage process: first, deep learning is used to identify text blocks in the image, and then text recognition technology is used to identify those blocks. This process suffers from low efficiency and high complexity.

[0044] In view of this, the embodiments of this application aim to propose a training method for a text recognition model. This method involves concatenating the location information of text regions in a sample image with the actual text within those regions to form a corresponding real label. The text recognition model is then trained using the sample image and its corresponding real label. The real label provides a supervisory signal for the training process, enabling the model to fully learn and understand the image features of the text regions in the sample image and the features of the actual text within those regions from the real label. This allows the model to simultaneously possess text detection and text recognition capabilities. Specifically, a text generation task is introduced. A first mask label is obtained by masking the actual text in the real label. The encoding module in the text recognition model encodes the sample image, and the decoding module in the model decodes the first image feature vector based on the first mask label to perform text recognition. The decoding module identifies the first character in the sample image and obtains the first character recognition result. Based on the real label corresponding to the sample image and the first character recognition result, the character recognition model is optimized. This enables the encoding module to accurately encode the relevant image features of the input image. Furthermore, since the unmasked content in the first mask label is the context related to the masked real text, the decoding module can not only learn the image features of the text region from the position information in the first mask label to master the knowledge of character detection, but also predict the masked real text by learning and understanding the semantic context of the masked real text. That is, it fully learns semantic information and the knowledge of text generation based on semantic information during the decoding process, and can then perform accurate character recognition based on the detected text region. In this way, the decoder has both character detection and character recognition capabilities, and the accuracy of both character detection and character recognition is high.

[0045] Based on this, this application also proposes a text recognition method. Since the encoder in the trained text recognition model has the ability to accurately encode the relevant image features of the input image, and the decoder in the text recognition model has the ability to both detect and recognize text, the encoding module in the trained text recognition model is used to encode the image to be recognized, and then the decoding module in the text recognition model is used to perform text recognition on the encoded image feature vector. This can achieve end-to-end text detection and recognition while ensuring the accuracy of text recognition, without the need for a two-stage algorithm that first performs text detection and then uses the relevant text recognition algorithm for text recognition. This simplifies the text recognition process and improves recognition efficiency.

[0046] It should be understood that the training method and the character recognition method of the character recognition model proposed in the embodiments of this application can both be executed by an electronic device or software installed in an electronic device. The electronic device referred to herein may include terminal devices, such as smartphones, tablets, laptops, desktop computers, smart voice interaction devices, smart home appliances, smartwatches, vehicle terminals, aircraft, etc.; or, the electronic device may also include a server, such as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0047] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0048] Please see Figure 1 The following is a flowchart illustrating a method for training a character recognition model, provided as an embodiment of this application. The method may include the following steps:

[0049] S102, Obtain the training dataset.

[0050] The training dataset includes sample images and their corresponding ground truth labels. The ground truth labels for the sample images are constructed by concatenating the location information of the text regions in the sample images with the actual text within those regions.

[0051] The text region in the sample image refers to the image region containing text. The positional information of the text region is used to indicate its location within the sample image. This positional information can be represented in various appropriate ways, and the specific method can be selected according to actual needs; this application does not limit this approach. Optionally, to accurately label the text region in the sample image, the positional information may include the pixel coordinates of the corner points of the text region, such as the pixel coordinates of the upper-left and lower-right corner points, i.e., (Xmin, Ymin, Xmax, Ymax), where Xmin represents the horizontal pixel coordinate of the upper-left corner point, Ymin represents the vertical pixel coordinate of the upper-left corner point, Xmax represents the horizontal pixel coordinate of the lower-right corner point, and Ymax represents the vertical pixel coordinate of the lower-right corner point.

[0052] In this embodiment, the real labels corresponding to the sample images can be constructed in various appropriate ways, and the specific method can be selected according to actual needs. This embodiment does not limit this method.

[0053] Optionally, after obtaining the pixel coordinates of the corner points of the text region in the sample image, the pixel coordinates of the corner points of the text region can be directly concatenated with the actual text in the text region. The concatenated result can then be used as the real label corresponding to the sample image. In practical applications, if there are multiple text regions in the sample image, for each text region, the pixel coordinates of the corner points of that text region can be concatenated with the actual text in that text region to obtain the sub-label corresponding to that text region. Then, the sub-labels corresponding to all text regions are concatenated, and a separator is set between adjacent sub-labels to obtain the real label corresponding to the sample image. For example, such as... Figure 2As shown, the boxes represent text regions in the sample image. Assuming the pixel coordinates of the corner points of the text region “Unifying Vision-and-Language Tasks via Text Generation” are (59, 25, 234, 89), and the pixel coordinates of the corner points of the text region “Jaemin ChoJie LeiHao TanMohitBansal” are (112, 102, 189, 136), then by concatenating the pixel coordinates of the corner points of these text regions with the actual text within those regions, we can obtain the actual label: 59 25 234 89 UnifyingVision – and – Language Tasks via Text Generation [sep] 112 102 189 136Jaemin Cho / t Jie Lei / t Hao Tan / t Mohit Bansal end. Here, / t represents a space, [sep] represents a separator, and end represents the end marker.

[0054] It is worth noting that in practical applications, the training dataset includes multiple sample images and their corresponding ground truth labels. For each sample image, the ground truth label can be constructed in the manner described above.

[0055] Optionally, for sample images in the training dataset, such as Figure 3 As shown, before performing subsequent steps, the sample images in the training dataset can be normalized to adjust them to a uniform size, such as 480×640 pixels, to facilitate subsequent processing.

[0056] S104, mask the real text in the real label to obtain the first mask label.

[0057] In this embodiment, masking the real text in the real label means hiding the real text within the real label. Thus, the resulting first masked label contains the context of the masked real text, and the relationship between the masked label and the masked real text reflects the semantics of the masked real text. In S104 above, the real label can be masked in various appropriate ways. Furthermore, during the masking process, since the real label contains both positional and textual information, it is necessary to map the real label into a vector that can be understood and processed by the character recognition model to facilitate its understanding.

[0058] Specifically, in one alternative implementation, such as Figure 3As shown, the location information and real text in the real label can be vector-mapped based on a preset dictionary to obtain the label vector corresponding to the real label; then, the values ​​in the label vector corresponding to the real text are masked to obtain the first mask label.

[0059] The preset dictionary records the mapping relationship between location information, real text, and values ​​within a preset range. During vector mapping, this mapping relationship is used to map the location information in the real label to the corresponding value, and the real text in the real label to the corresponding value. Further, these values ​​are concatenated sequentially to obtain the label vector corresponding to the real label. This label vector is the label form that can be understood and processed by the text recognition model. Finally, the values ​​corresponding to the real text that need to be masked in the label vector are masked, for example, by replacing the real text that needs to be masked with the preset mask character [MASK], to obtain the first mask label.

[0060] In this embodiment, the preset dictionary can be pre-constructed based on the position information of the text region of each sample image in the training dataset and the real text in the text region of each sample image. The preset dictionary obtained in this way can cover the mapping relationship between various position coordinates, real text and numerical values, which is conducive to improving the accuracy of vector mapping.

[0061] More specifically, based on the location information of the text regions in each sample image in the training dataset and the actual text in the text regions of each sample image, a preset dictionary is constructed, which may include: dividing a preset numerical range into a first numerical range and a second numerical range based on the image size of each sample image in the training dataset, wherein the first numerical range is used to map location information and the second numerical range is used to map text; constructing a first mapping relationship between location information and values ​​in the first numerical range based on the first numerical range and the location information of the text regions in each sample image, and constructing a second mapping relationship between actual text and values ​​in the second numerical range based on the second numerical range and the actual text in the text regions of each sample image; and constructing a preset dictionary based on the first mapping relationship and the second mapping relationship.

[0062] For example, such as Figure 4As shown, assuming the image size of each sample image in the training dataset is 480×640, the maximum pixel coordinate value of the text region in these sample images can be considered to be 640. Therefore, 640 can be used as a boundary value, 0~639 can be used as the first numerical range for mapping positional information, and 640~n can be used as the second numerical range for mapping text, where n is an integer greater than 640. The specific value can be set according to actual needs, such as n being ∞. Thus, a first mapping relationship can be established between positional information (such as pixel coordinates) and values ​​within the first numerical range, and a second mapping relationship can be established between real text and values ​​within the second numerical range.

[0063] For example, with Figure 4 Taking the first and second preset mapping ranges as examples, assuming the real label is "59 25 234 89 Unifying Vision – and – Language Tasks via Text Generation", the position information in the real label and the real text are vector-mapped based on the mapping relationship recorded in the preset dictionary, resulting in the corresponding label vector [59 25 234 89 641 678 850 832 850 753 654 1023 2045 1889]. Here, 59, 25, 234, and 89 in the label vector correspond to the pixel coordinates 59, 25, 234, and 89 in the real label, respectively, and 641 represents the real text "Unifying". 678 represents the real text "Vision", 850 represents the real text "-", 832 represents the real text "and", 753 represents the real text "Language", 654 represents the real text "Tasks", 1023 represents the real text "via", 2045 represents the real text "Text", and 1889 represents the real text "Generation".

[0064] It is understood that the dictionary construction method in this application embodiment divides the sample images in the training dataset into bins, for example, dividing them into two numerical ranges with 640 as the boundary value. One numerical range is used to map position information, and the other numerical range is used to map text. The dictionary constructed in this way includes both position information and text. On this basis, the position information and text information in the real labels are vector-mapped using a preset dictionary. This not only ensures that the mapped label vectors completely retain the position information and text of the text region, but also enables the label vectors to be understood and processed by the text recognition model, thereby improving the training effect of the text recognition model.

[0065] After obtaining the label vector, the values ​​corresponding to the real text in the label vector can be masked in various appropriate ways to obtain the first mask label. This application embodiment does not limit this.

[0066] Optionally, the first mask label can be obtained as follows: the numerical values ​​corresponding to the real text in the label vector can be masked using a preset mask character, and the masked label vector can be used as the first mask label. For example, continuing with the label vector [59 25 234 89 641 678 850 832 850 753 654 1023 20451889] shown above, assuming the real text to be masked is "Text", then by replacing the real text with the preset mask character [MASK], the first mask label [59 25 234 89 641 678 850 832 850 753 654 1023[MASK] 1889] can be obtained.

[0067] Optionally, such as Figure 3 As shown, after obtaining the label vector, it can be enhanced, and then the enhanced candidate label vector can be masked to obtain the first mask label. Specifically, combined with Figure 5 As shown, the first mask label can be obtained as follows: the label vector is enhanced to obtain a candidate label vector; then, the values ​​corresponding to some real text in the candidate label vector are masked to obtain the first mask label. The enhancement process includes at least one of the following processes: value deletion, value replacement, and value insertion. Value deletion refers to deleting some values ​​in the label vector, value replacement refers to replacing some values ​​in the label vector with other values ​​or characters, and value insertion refers to inserting other values ​​into the label vector.

[0068] For example, continuing with the label vector [59 25 234 89 641 678 850 832 850 753 654 1023 2045 1889] shown above, we can randomly select some values ​​from this label vector and delete them with a certain probability. For example, we can select 234 to delete, thus obtaining the candidate label vector [59 25 89 641 678 850 832 850 753 654 1023 2045 1889]. Furthermore, we can use the preset mask character [MASK] to replace some of the values ​​corresponding to the real text for masking processing. For example, 59 can be used to obtain the first mask label [[MASK] 25 89 641 678 850 832 850 753 654 1023 2045 1889]. This approach can simulate the situation where some text is missing in a text region, which helps improve the recognition effect of the text recognition model on images with missing text. This enhances both the robustness and error correction capabilities of the text recognition model.

[0069] Alternatively, a portion of the values ​​can be randomly selected from the label vector with a certain probability for replacement. For example, 234 can be replaced with the noise character "n / a", resulting in the candidate label vector [59 25 n / a 89 641 678 850 832 850 753 6541023 2045 1889]. Further, a masking process is performed by replacing some of the values ​​corresponding to the real text with a preset mask character [MASK]. For example, 59 yields the first mask label [[MASK] 25 n / a 89 641 678 850 832 850 753 6541023 2045 1889]. This simulates situations where some text in a text area is blurred or misspelled, which helps improve the recognition effect of the text recognition model on images with blurred or misspelled text. This enhances both the robustness and error correction capabilities of the text recognition model.

[0070] Alternatively, other characters can be randomly inserted into the label vector. For example, selecting 231 yields the candidate label vector [59 25 234 231 89 641 678 850 832 850 753 654 1023 2045 1889]. Further, a preset mask character [MASK] is used to replace the numerical values ​​corresponding to some of the real text for masking. For example, 59 yields the first mask label [[MASK] 25 234 231 89 641 678 850 832 850 753 654 1023 2045 1889]. This simulates situations where some text in a text area is blurred or misspelled, which helps improve the recognition effect of the text recognition model on images with blurred or misspelled text. This enhances both the robustness and error correction capabilities of the text recognition model.

[0071] Alternatively, at least two enhancement processes can be performed on the above label vectors to obtain candidate label vectors; and the values ​​in the candidate label vectors that correspond to some of the real text can be masked to obtain the first masked label.

[0072] It is understandable that by performing at least one label enhancement process on the label vector, such as deleting text, replacing text, or inserting text, the enhanced candidate label vector carries noise. On this basis, a masking process is performed, and the resulting first mask label can simulate at least one of the following situations: blurred text in the image, missing text, and misspelling. This is beneficial to improving the text recognition effect of the text recognition model, thereby improving the robustness of the text recognition model and its text error correction capability.

[0073] It is worth noting that when masking the label vector, the text following the text to be predicted can also be masked. For example, assuming the real text is "I love China", when predicting the character "love", the value corresponding to "China" in the label vector can be masked, so that the text recognition model can only see the value corresponding to the character "I" before "love". This helps to prevent the real text in the real label from being leaked.

[0074] In another alternative implementation, a portion of the real text in the real label can be masked first, for example, by replacing some of the real text with preset mask characters to obtain candidate mask labels. Then, the position information, preset mask characters, and unmasked real text in the candidate mask labels are mapped to corresponding numerical values ​​to obtain the first mask label. In this case, the preset dictionary can also record the mapping relationship between preset mask characters and numerical values.

[0075] This application embodiment illustrates a partial specific implementation of S104 described above. It should be understood that S104 can also be implemented in other ways, and this application embodiment does not limit this implementation.

[0076] S106, The sample image is encoded by the encoding module in the character recognition model to obtain the first image feature vector of the sample image.

[0077] The encoding module refers to the module that can encode the input image. It can adopt various image encoding models commonly used in the field, such as the Transformer encoder. The specific structure can be set according to actual needs, and the embodiments of this application do not limit it.

[0078] Encoding a sample image refers to mapping the image features of the sample image into a vector that can be recognized by an electronic device that performs the training method for the character recognition model; this is known as the first image feature vector. In other words, the first image feature vector is used to represent the image features of the sample image.

[0079] In S106 above, optionally, the sample image can be directly input into the encoding module to obtain the first image feature vector of the sample image. Optionally, considering that some encoding modules are not suitable for encoding the entire image, such as the Transformer encoder, in this case, such as... Figure 3 As shown, the sample image can first be divided into blocks. For example, a sample image of size 480×640 pixels can be divided into 16 blocks, and the size of each image block is 30×40 pixels. Then, each image block is encoded. For example, each image block is linearly mapped to a vector space of a preset dimension (e.g., 768 dimensions). For each image block, the mapping result corresponding to the image block is added to the position encoding information used to represent the position information of the image block to obtain the embedding vector corresponding to the image block. Further, each image is encoded by the encoding module to obtain the first image feature vector corresponding to the sample image.

[0080] Optionally, the sample image may also undergo image enhancement processing; then, the enhanced sample image is encoded by an encoding module. The image enhancement processing includes at least one of the following: random rotation, image stretching, contrast enhancement, color enhancement, Gaussian blur, image dilation, image erosion, downsampling, adding underlines, etc.

[0081] In practical applications, image enhancement processing can be performed at any appropriate time, and the specific settings can be configured according to actual needs. For example, such as... Figure 3 As shown, image enhancement processing can be performed on the sample image after normalization and before image segmentation.

[0082] It is understandable that by performing the above image enhancement processing on the sample images and then encoding them, the training process can simulate the deformed and blurred text recognition scenario, thereby improving the accuracy of the text recognition model in recognizing deformed and blurred text in real-world scenarios. This eliminates the need for the traditional text recognition step of detecting and recognizing text before correction, thus improving recognition efficiency.

[0083] S108, the decoding module in the character recognition model performs character recognition on the feature vector of the first image based on the first mask label, and obtains the first character recognition result corresponding to the sample image.

[0084] The decoding module may include various modules with information decoding capabilities, such as the Transformer decoder. The specific module can be selected according to actual needs, and this application embodiment does not limit this.

[0085] The first character recognition result corresponding to the sample image can include the first predicted character region of the sample image and the predicted character within the first predicted character region. In practical applications, the first character recognition result can be a multi-dimensional vector, which includes the numerical values ​​corresponding to the positional information of the first predicted character region and the numerical values ​​corresponding to the predicted character within the first predicted character region.

[0086] S110 optimizes the text recognition model based on the real labels corresponding to the sample images and the first text recognition result.

[0087] Specifically, the recognition loss of the character recognition model can be calculated based on the real labels corresponding to the sample images and the first character recognition result. Then, based on the recognition loss of the character recognition model, the model parameters of the character recognition model can be adjusted, for example, by using gradient descent algorithms, backpropagation algorithms, etc. The model parameters of the character recognition model can include the model parameters of the encoding module and the model parameters of the decoding module.

[0088] It is worth noting that the above is only a single training iteration. In practical applications, steps S104 to S110 can be executed multiple times until the character recognition model meets the preset training stopping conditions. These preset training stopping conditions may include: the character recognition model's recognition loss is less than a preset loss threshold, or the character recognition model's recognition accuracy is higher than a preset accuracy threshold, or the number of training iterations reaches a preset number threshold, etc. These conditions can be set according to actual needs, and this application embodiment does not limit them. Secondly, the character recognition model used in the first iteration of training can be called the initial character recognition model. It can be an existing character recognition model with preliminary character recognition capabilities, or it can be a character recognition model that does not yet possess character recognition capabilities. From the second iteration onwards, each iteration optimizes the character recognition model based on the previous iteration, resulting in a character recognition model different from the previous iteration. After multiple iterations of training, the character recognition model that meets the preset training stopping conditions can be used for subsequent character recognition.

[0089] Optionally, in another embodiment of this application, after S108 above, the training method of the character recognition model provided in this application embodiment can also introduce error correction for the character recognition model, so that the character recognition model has the ability to delete duplicate and useless characters and correct incomplete output results, thereby further improving the robustness and accuracy of the character recognition model and enhancing the character error correction capability of the character recognition model.

[0090] Specifically, in combination Figure 6 As shown, fine-tuning the character recognition model includes the following steps:

[0091] First, the first character recognition result is used as the initial training label. The training label and the corresponding masked preset error correction character are concatenated. The predicted characters in the first character recognition result in the concatenated result are then masked to obtain the second mask label corresponding to the sample image.

[0092] It is worth noting that the process of obtaining the second mask label described above is similar to... Figure 3 The process of obtaining the first mask label in the illustrated embodiment is similar. For example, the training label and the preset error correction symbol after the mask are concatenated, the concatenation result is vector mapped to obtain the corresponding label vector, and then the corresponding candidate label vector is obtained through label enhancement processing. Finally, the second mask label is obtained through mask processing. The details will not be repeated.

[0093] In practical applications, when concatenating the training labels with the preset error correction character after the mask, the preset error correction character after the mask can be concatenated with the preset text "The required modification operation is:" to obtain the following candidate concatenation result: "The required modification operation is:"<mask>", in, <mask>This represents the preset error correction character after the mask; then, the training label is concatenated with this candidate concatenation result, finally obtaining the concatenation result in the following format: The modification operation required for the training label [sep] is as follows: <mask>Where [sep] represents the separator. For example, assuming the training labels are [59 25 234 89 641678 850 832 850 753 654 1023 2045 1889], the final concatenation result is: 59 25 234 89641 678 850 832 850 753 654 1023 2045 1889 [sep]. The necessary modification is: <mask>In this context, "59 25 234 89" represents the pixel coordinates of the corner point of the predicted text region in the training label as (59 25 234 89), and "641 678 850 832 850 753 654 1023 2045 1889" represents the predicted text in the predicted text region.

[0094] The preset error correction symbol corresponding to the training label is used to indicate the type of target error correction operation to be performed on the predicted text in the training label. For example, the preset error correction symbol can include three types: 0, 1, and 2. 0 indicates that the target error correction operation is no correction, 1 indicates that the target error correction operation is inserting text, and 2 indicates that the target error correction operation is deleting text.

[0095] Optionally, to improve the text recognition model's error correction capability, the preset error correction symbol can be obtained as follows: compare the length of the predicted text in the training labels with the length of the real text in the true labels, and determine the preset error correction symbol corresponding to the training labels based on the comparison result. For example, if the length of the predicted text in the training labels is equal to the length of the real text in the true labels, it can be preliminarily determined that the text in the two labels matches, and thus the preset error correction symbol can be determined to be 0; if the length of the predicted text in the training labels is less than the length of the real text in the true labels, it can be preliminarily determined that the predicted text in the training labels is missing compared to the real text, and thus the preset error correction symbol can be determined to be 1; if the length of the predicted text in the training labels is greater than the length of the real text in the true labels, it can be preliminarily determined that the predicted text in the training labels is increased compared to the real text, and thus the preset error correction symbol can be determined to be 2.

[0096] Next, the sample image is encoded by the encoding module to obtain the second image feature vector of the sample image, and the second image feature vector is then used by the decoding module to perform text recognition based on the second mask label to obtain the second text recognition result corresponding to the sample image.

[0097] The second text recognition result includes the second predicted text region of the sample image, the predicted text in the second predicted text region, and the first predicted error correction symbol corresponding to the preset error correction symbol. The first predicted error correction symbol is used to indicate the operation type of the prediction error correction operation that needs to be performed on the predicted text in the predicted label.

[0098] It is worth noting that, such as Figure 6 As shown, during the fine-tuning process, the process of encoding the sample image through the encoding module is the same as described above. Figure 3 The encoding process for the sample image in the illustrated embodiment is similar and may also include normalization, image enhancement, image block processing, linear mapping, addition with the positional encoding result, and encoding by the encoding module, etc., which will not be elaborated further. Additionally, the decoding module performs text recognition on the second image feature vector based on the second mask label, which is similar to the process of performing text recognition on the first image feature vector based on the first mask label, and will not be elaborated further.

[0099] Furthermore, the encoding module is fine-tuned based on the second predicted text region of the sample image, the predicted text in the second predicted text region, and the real label, and the decoding module is fine-tuned based on the operation type of the first predicted error correction symbol and the target error correction operation.

[0100] For example, the text recognition loss of the text recognition model can be calculated based on the second predicted text region of the sample image, the predicted text in the second predicted text region, and the real label. Based on the text recognition loss, the model parameters of the encoding module can be adjusted. The text recognition loss is used to represent the difference between the positional information in the second predicted text region and the real label based on the sample image, as well as the difference between the predicted text in the second predicted text region and the real text in the real label.

[0101] Similarly, based on the operation types represented by the first predicted error correction symbol and the preset error correction symbol, the error correction loss of the character recognition model is calculated, and the model parameters of the decoding module are adjusted based on the error correction loss. The error correction loss is used to represent the difference between the operation type represented by the first predicted error correction symbol and the operation type represented by the preset error correction symbol.

[0102] Finally, based on the second predicted text region of the sample image and the predicted text in the second predicted text region, a new training label is determined. The above steps of concatenating the training label and the preset error correction symbol after the mask to fine-tune the decoding module are repeated until the preset fine-tuning stop condition is met.

[0103] It is worth noting that the new training label is determined based on the second predicted text region of the sample image and the predicted text in the second predicted text region, which is consistent with the above. Figure 1 The implementation method for obtaining the real label corresponding to the sample image in the illustrated embodiment is similar and will not be described again.

[0104] In addition, the preset fine-tuning stopping condition can be set according to actual needs, such as the number of fine-tuning iterations reaching a preset threshold, etc., which is not limited in this embodiment.

[0105] The text recognition model training method provided in one or more embodiments of this application concatenates the location information of text regions in a sample image with the real text in those regions to form a real label corresponding to the sample image. The text recognition model is then trained using the sample image and its corresponding real label. The real label provides a supervisory signal for the training process, enabling the text recognition model to fully learn and understand the image features of the location of the text regions in the sample image and the features of the real text within those regions from the real label, thus simultaneously possessing the ability to detect and recognize text. Specifically, a text generation task is introduced. A first mask label is obtained by masking the real text in the real label. The sample image is then encoded by the encoding module in the text recognition model, and the text recognition model's decoding module performs text recognition based on the first mask label on the encoded first image feature vector to obtain the text corresponding to the sample image. The first character recognition result, along with the corresponding real label and the first character recognition result based on the sample image, optimizes the character recognition model. This enables the encoding module to accurately encode relevant image features of the input image. Furthermore, the first mask label obtained after masking effectively prevents label information leakage. Since the unmasked content in the first mask label is context related to the masked real text, the decoding module can not only learn the image features of the text region's location from the positional information in the first mask label to master character detection knowledge, but also predict the masked real text by learning and understanding the semantic context of the masked real text. In other words, it fully learns semantic information and knowledge of text generation based on semantic information during the decoding process, thus enabling accurate character recognition based on the detected text region. Therefore, the decoder simultaneously possesses the ability to detect and recognize characters, with high accuracy in both.

[0106] Based on the character recognition model trained as described above, this application also provides a character recognition method. Please refer to... Figure 7 The following is a flowchart illustrating a character recognition method according to an embodiment of this application. The method may include the following steps:

[0107] S702, acquire the image to be recognized.

[0108] S704 uses the encoding module in the character recognition model to encode the image to be recognized, thereby obtaining the image feature vector of the image to be recognized.

[0109] Optionally, the process of encoding the image to be recognized is the same as described above. Figure 3 The process of encoding the sample image in the illustrated embodiment is similar and may also include normalization of the image to be recognized, image enhancement, image block processing, linear mapping, addition with the position encoding result, and encoding through the encoding module, etc., which will not be described in detail here.

[0110] S706 uses the decoding module in the character recognition model to perform character recognition on the image feature vector, and obtains the character recognition result corresponding to the image to be recognized.

[0111] Optionally, combined Figure 8 The process of performing character recognition on the image feature vector through the decoding module in the character recognition model to obtain the character recognition result corresponding to the image to be recognized may include the following steps:

[0112] First, the image feature vector and the preset mask characters are concatenated to obtain the concatenation result.

[0113] For example, a preset mask character can be concatenated with the preset text "The modification operation to be performed is:" to obtain the following candidate concatenation result: "The modification operation to be performed is:" <mask>",in, <mask>This represents the preset mask character; then, the image feature vector is concatenated with this candidate concatenation result, finally obtaining the concatenation result in the following format: The image feature vector [sep] needs to be modified as follows: <mask>.

[0114] Next, the decoding module performs text recognition on the spliced ​​result to obtain the text recognition result corresponding to the image to be recognized.

[0115] The text recognition result includes: the predicted text region of the image to be recognized, the predicted text in the predicted text region, and the second prediction correction character corresponding to the preset mask character. The second prediction correction character is used to indicate the operation type of the prediction correction operation to be performed on the predicted text in the predicted text region.

[0116] Furthermore, if the operation type of the predicted error correction operation is an operation other than no error correction, such as text deletion or text insertion, then the image feature vector, the preset mask character, the predicted text region of the image to be recognized, and the predicted text in the predicted text region are concatenated to obtain a new concatenation result. The above steps of performing text recognition on the concatenation result through the decoding module to concatenate the image feature vector, the preset mask character, the predicted text region of the image to be recognized, and the predicted text in the predicted text region are repeated until the operation type represented by the second predicted error correction symbol is no error correction.

[0117] The text recognition method provided in one or more embodiments of this application has the ability to accurately encode the relevant image features of the input image by means of the encoder in the trained text recognition model, and the decoder in the text recognition model has the ability to both detect and recognize text. By using the encoding module in the trained text recognition model to encode the image to be recognized, and then using the decoding module in the text recognition model to recognize text by the encoded image feature vector, end-to-end text detection and recognition can be achieved while ensuring the accuracy of text recognition, without the need for a two-stage algorithm that first detects text and then uses a relevant text recognition algorithm to recognize text. This simplifies the text recognition process and improves recognition efficiency.

[0118] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0119] In addition, with the above Figure 1 Corresponding to the training method of the character recognition model shown, this application embodiment also provides a training device for a character recognition model. Please refer to... Figure 9 The diagram below illustrates the structure of a text classification model training device 900 according to an embodiment of this application. The device 900 may include:

[0120] The first acquisition unit 910 is used to acquire a training dataset, which includes sample images and corresponding real labels for the sample images. The real labels are formed by splicing the position information of the text region in the sample image and the real text in the text region.

[0121] The first masking unit 920 is used to mask the real text in the real label to obtain the first masked label.

[0122] The first recognition unit 930 is used to encode the sample image through the encoding module in the character recognition model to obtain the first image feature vector of the sample image, and to perform character recognition on the first image feature vector based on the first mask label through the decoding module in the character recognition model to obtain the first character recognition result corresponding to the sample image.

[0123] The first optimization unit 940 is used to optimize the text recognition model based on the real label corresponding to the sample image and the first text recognition result.

[0124] Optionally, the first masking unit performs masking processing on the real text in the real label to obtain a first masked label, including:

[0125] Based on a preset dictionary, the location information and real text in the real labels are vector-mapped to obtain the label vector corresponding to the real labels. The preset dictionary is constructed based on the location information of the text region of each sample image in the training dataset and the real text in the text region of each sample image. The preset dictionary is used to represent the mapping relationship between location information, real text and values ​​within a preset numerical range.

[0126] The numerical values ​​in the label vector corresponding to the real text are masked to obtain the first masked label.

[0127] Optionally, the preset dictionary is constructed in the following manner:

[0128] Based on the image size of each sample image in the training dataset, the preset numerical range is divided into a first numerical range and a second numerical range. The first numerical range is used to map location information, and the second numerical range is used to map text.

[0129] Based on the first numerical range and the location information of the text region in each sample image, a first mapping relationship is constructed between the location information and the numerical value within the first numerical range.

[0130] Based on the second numerical range and the real text in the text region of each sample image, a second mapping relationship is constructed between the real text and the numerical values ​​within the second numerical range;

[0131] Based on the first mapping relationship and the second mapping relationship, the preset dictionary is constructed.

[0132] Optionally, the first masking unit performs masking processing on the numerical values ​​corresponding to the real text in the label vector to obtain the first mask label, including:

[0133] The label vector is enhanced to obtain a candidate label vector. The enhancement process includes at least one of the following processes: numerical deletion, numerical replacement, and numerical insertion.

[0134] The numerical values ​​corresponding to some of the real text in the candidate label vector are masked to obtain the first masked label.

[0135] Optionally, the first recognition unit encodes the sample image using the encoding module in the character recognition model, including:

[0136] The sample image is subjected to image enhancement processing;

[0137] The sample image after image enhancement is encoded using the encoding module in the text recognition model.

[0138] Optionally, the first character recognition result includes a first predicted character region of the sample image and the predicted characters in the first predicted character region;

[0139] The first optimization unit is further configured to:

[0140] After adjusting the model parameters of the text recognition model based on the real label corresponding to the sample image and the first text recognition result, the first text recognition result is used as the initial training label. The training label and the preset error correction symbol corresponding to the training label after masking are concatenated, and the predicted text in the first text recognition result in the concatenation result is masked to obtain the second mask label corresponding to the sample image. The preset error correction symbol is used to indicate the operation type of the target error correction operation to be performed on the predicted text in the training label.

[0141] The sample image is encoded by the encoding module to obtain a second image feature vector of the sample image. The second image feature vector is then used by the decoding module to perform text recognition based on the second mask label to obtain a second text recognition result corresponding to the sample image. The second text recognition result includes a second predicted text region of the sample image, the predicted text in the second predicted text region, and a first prediction error correction symbol corresponding to the preset error correction symbol. The first prediction error correction symbol is used to indicate the operation type of the prediction error correction operation to be performed on the predicted text in the training label.

[0142] Based on the second predicted text region of the sample image, the predicted text in the second predicted text region, and the real label, the encoding module is fine-tuned.

[0143] Based on the operation type of the first predicted error corrector and the target error correction operation, the decoding module is fine-tuned;

[0144] Based on the second predicted text region of the sample image and the predicted text in the second predicted text region, a new training label is determined. The above steps of concatenating the training label and the preset error correction symbol after the mask to fine-tune the decoding module are repeated until the preset fine-tuning stop condition is met.

[0145] Optionally, the first optimization unit is further configured to: before concatenating the training label and the preset error correction character corresponding to the training label after the mask, compare the length of the predicted text in the training label with the length of the real text in the real label;

[0146] Based on the comparison results, the preset error correction symbol corresponding to the training label is determined.

[0147] Obviously, the training device for the character recognition model provided in this application embodiment can be used as described above. Figure 1 The main body executing the training method of the shown character recognition model is thus able to realize the training method of the character recognition model in... Figure 9 The functions implemented are the same, so I will not go into details.

[0148] In addition, with the above Figure 7 Corresponding to the text recognition method shown, this application also provides a text recognition device. Please refer to... Figure 10 The image below is a schematic diagram of the structure of a character recognition device 1000 provided in one embodiment of this application. The device 1000 may include:

[0149] The second acquisition unit 1010 is used to acquire the image to be recognized;

[0150] The second recognition unit 1020 is used to encode the image to be recognized through the encoding module in the character recognition model to obtain the image feature vector of the image to be recognized, and to perform character recognition on the image feature vector through the decoding module in the character recognition model to obtain the character recognition result corresponding to the image to be recognized. The character recognition model is trained based on the training method of the character recognition model described in the embodiments of this application.

[0151] Optionally, the second recognition unit performs character recognition on the image feature vector through the decoding module in the character recognition model to obtain the character recognition result corresponding to the image to be recognized, including:

[0152] The image feature vector and the preset mask characters are concatenated to obtain the concatenation result;

[0153] The decoding module performs text recognition on the splicing result to obtain the text recognition result corresponding to the image to be recognized. The text recognition result includes: the predicted text region of the image to be recognized, the predicted text in the predicted text region, and the second prediction error correction character corresponding to the preset mask character. The second prediction error correction character is used to indicate the operation type of the prediction error correction operation that needs to be performed on the predicted text in the predicted text region.

[0154] If the operation type represented by the second prediction error correction symbol is a text deletion operation or a text insertion operation, then the image feature vector, the preset mask character, the predicted text region of the image to be recognized, and the predicted text in the predicted text region are concatenated to obtain a new concatenation result. The above steps of performing text recognition on the concatenation result through the decoding module up to concatenating the image feature vector, the preset mask character, the predicted text region of the image to be recognized, and the predicted text in the predicted text region are repeated until the operation type represented by the obtained second prediction error correction symbol is no error correction.

[0155] Obviously, the character recognition device provided in the embodiments of this application can be used as described above. Figure 7 The shown text recognition method is the execution subject, thus enabling the text recognition method to perform... Figure 7 The functions implemented are the same, so I will not go into details.

[0156] Figure 11 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Please refer to it. Figure 11 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and memory. The memory may include main memory, such as high-speed random-access memory (RAM), or non-volatile memory, such as at least one disk drive. Of course, the electronic device may also include other hardware required for other business operations.

[0157] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 11 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0158] Memory is used to store programs. Specifically, programs may include program code, which includes computer operation instructions. Memory may include main memory and non-volatile memory, and provides instructions and data to the processor.

[0159] The processor reads the corresponding computer program from non-volatile memory into main memory and then runs it, forming a training device for the character recognition model at the logical level. The processor executes the program stored in memory and specifically performs the following operations:

[0160] Obtain a training dataset, which includes sample images and corresponding real labels for the sample images. The real labels are formed by concatenating the positional information of the text regions in the sample images with the real text in the text regions.

[0161] The real text in the real label is masked to obtain the first masked label;

[0162] The sample image is encoded by the encoding module in the character recognition model to obtain the first image feature vector of the sample image. The character recognition model is then decoded by the decoding module based on the first mask label to perform character recognition on the first image feature vector to obtain the first character recognition result corresponding to the sample image.

[0163] Based on the real labels corresponding to the sample images and the first character recognition result, the character recognition model is optimized.

[0164] Alternatively, the processor reads the corresponding computer program from non-volatile memory into memory and runs it, forming a character recognition device at the logical level. The processor executes the program stored in memory and specifically performs the following operations:

[0165] Acquire the image to be recognized;

[0166] The image to be recognized is encoded by the encoding module in the character recognition model to obtain the image feature vector of the image to be recognized, and the character recognition is performed on the image feature vector by the decoding module in the character recognition model to obtain the character recognition result corresponding to the image to be recognized. The character recognition model is trained based on the training method of the character recognition model described in the embodiments of this application.

[0167] The above is as stated in this application. Figure 1 The illustrated embodiment discloses a method for training a character recognition model using a device, or the method described above in this application. Figure 7 The method executed by the character recognition device disclosed in the illustrated embodiments can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0168] The electronic device can also perform Figure 1 The method, and the implementation of the text recognition model training device in Figures 1 to 6 The illustrated embodiment may also perform the functions of the electronic device, or the electronic device may also perform the functions of the embodiment shown. Figure 7 The method, and to realize the character recognition device in Figure 7 , Figure 8 The functions of the embodiments shown are not described again in this application.

[0169] Of course, in addition to software implementation, the electronic device of this application does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. In other words, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0170] This application also proposes a computer-readable storage medium that stores one or more programs, the programs including instructions that, when executed by a portable electronic device including multiple applications, enable the portable electronic device to perform... Figure 1 The method of the illustrated embodiment is specifically used to perform the following operations:

[0171] Obtain a training dataset, which includes sample images and corresponding real labels for the sample images. The real labels are formed by concatenating the positional information of the text regions in the sample images with the real text in the text regions.

[0172] The real text in the real label is masked to obtain the first masked label;

[0173] The sample image is encoded by the encoding module in the character recognition model to obtain the first image feature vector of the sample image. The character recognition model is then decoded by the decoding module based on the first mask label to perform character recognition on the first image feature vector to obtain the first character recognition result corresponding to the sample image.

[0174] Based on the real labels corresponding to the sample images and the first character recognition result, the character recognition model is optimized.

[0175] Alternatively, when executed by a portable electronic device that includes multiple applications, the instruction can enable the portable electronic device to perform... Figure 7 The method of the illustrated embodiment is specifically used to perform the following operations:

[0176] Acquire the image to be recognized;

[0177] The image to be recognized is encoded by the encoding module in the character recognition model to obtain the image feature vector of the image to be recognized, and the character recognition is performed on the image feature vector by the decoding module in the character recognition model to obtain the character recognition result corresponding to the image to be recognized. The character recognition model is trained based on the training method of the character recognition model described in the embodiments of this application.

[0178] In summary, the above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

[0179] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0180] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0181] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0182] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.< / mask> < / mask> < / mask> < / mask> < / mask> < / mask> < / mask>

Claims

1. A method for training a character recognition model, characterized in that, include: Obtain a training dataset, which includes sample images and corresponding real labels for the sample images. The real labels are formed by concatenating the positional information of the text regions in the sample images with the real text in the text regions. The real text in the real label is masked to obtain the first masked label; The sample image is encoded by the encoding module in the character recognition model to obtain a first image feature vector of the sample image. Then, the character recognition model's decoding module performs character recognition on the first image feature vector based on the first mask label to obtain a first character recognition result corresponding to the sample image. The first character recognition result includes a first predicted text region of the sample image and the predicted text in the first predicted text region. The predicted text is generated by predicting and generating the masked real text based on the first image feature vector and the unmasked content in the first mask label. Based on the real labels corresponding to the sample images and the first character recognition result, the character recognition model is optimized.

2. The method according to claim 1, characterized in that, The process of masking the real text in the real label to obtain the first masked label includes: Based on a preset dictionary, the location information and real text in the real labels are vector-mapped to obtain the label vector corresponding to the real labels. The preset dictionary is constructed based on the location information of the text region of each sample image in the training dataset and the real text in the text region of each sample image. The preset dictionary is used to represent the mapping relationship between location information, real text and values ​​within a preset numerical range. The numerical values ​​in the label vector corresponding to the real text are masked to obtain the first masked label.

3. The method according to claim 2, characterized in that, The preset dictionary is constructed in the following manner: Based on the image size of each sample image in the training dataset, the preset numerical range is divided into a first numerical range and a second numerical range. The first numerical range is used to map location information, and the second numerical range is used to map text. Based on the first numerical range and the location information of the text region in each sample image, a first mapping relationship is constructed between the location information and the numerical value within the first numerical range. Based on the second numerical range and the real text in the text region of each sample image, a second mapping relationship is constructed between the real text and the numerical values ​​within the second numerical range; Based on the first mapping relationship and the second mapping relationship, the preset dictionary is constructed.

4. The method according to claim 2, characterized in that, The step of masking the values ​​corresponding to the real text in the label vector to obtain the first masked label includes: The label vector is enhanced to obtain a candidate label vector. The enhancement process includes at least one of the following processes: numerical deletion, numerical replacement, and numerical insertion. The numerical values ​​corresponding to some of the real text in the candidate label vector are masked to obtain the first masked label.

5. The method according to claim 1, characterized in that, The encoding process of the sample image through the encoding module in the character recognition model includes: The sample image is subjected to image enhancement processing; The sample image after image enhancement is encoded using the encoding module in the text recognition model.

6. The method according to claim 1, characterized in that, The first character recognition result includes the first predicted character region of the sample image and the predicted character in the first predicted character region; After adjusting the model parameters of the character recognition model based on the real labels corresponding to the sample images and the first character recognition result, the method further includes: The first character recognition result is used as the initial training label. The training label and the preset error correction symbol corresponding to the training label after masking are concatenated. The predicted characters in the first character recognition result in the concatenation result are masked to obtain the second mask label corresponding to the sample image. The preset error correction symbol is used to indicate the operation type of the target error correction operation to be performed on the predicted characters in the training label. The sample image is encoded by the encoding module to obtain a second image feature vector of the sample image. The second image feature vector is then used by the decoding module to perform text recognition based on the second mask label to obtain a second text recognition result corresponding to the sample image. The second text recognition result includes a second predicted text region of the sample image, the predicted text in the second predicted text region, and a first prediction error correction symbol corresponding to the preset error correction symbol. The first prediction error correction symbol is used to indicate the operation type of the prediction error correction operation to be performed on the predicted text in the training label. Based on the second predicted text region of the sample image, the predicted text in the second predicted text region, and the real label, the encoding module is fine-tuned. Based on the operation type of the first predicted error corrector and the target error correction operation, the decoding module is fine-tuned; Based on the second predicted text region of the sample image and the predicted text in the second predicted text region, a new training label is determined. The above steps of concatenating the training label and the preset error correction symbol after the mask to fine-tune the decoding module are repeated until the preset fine-tuning stop condition is met.

7. The method according to claim 6, characterized in that, Before concatenating the training labels and the corresponding preset error correction symbols after the mask, the method further includes: The length of the predicted text in the training labels is compared with the length of the real text in the real labels; Based on the comparison results, the preset error correction symbol corresponding to the training label is determined.

8. A character recognition method, characterized in that, include: Acquire the image to be recognized; The image to be recognized is encoded by the encoding module in the character recognition model to obtain the image feature vector of the image to be recognized, and the character recognition is performed on the image feature vector by the decoding module in the character recognition model to obtain the character recognition result corresponding to the image to be recognized. The character recognition model is trained based on the training method of the character recognition model according to any one of claims 1 to 7.

9. The method according to claim 8, characterized in that, The character recognition model is trained using the training method described in claim 6 or 7. The step of performing text recognition on the image feature vector through the decoding module in the text recognition model to obtain the text recognition result corresponding to the image to be recognized includes: The image feature vector and the preset mask characters are concatenated to obtain the concatenation result; The decoding module performs text recognition on the splicing result to obtain the text recognition result corresponding to the image to be recognized. The text recognition result includes: the predicted text region of the image to be recognized, the predicted text in the predicted text region, and the second prediction error correction character corresponding to the preset mask character. The second prediction error correction character is used to indicate the operation type of the prediction error correction operation that needs to be performed on the predicted text in the predicted text region. If the operation type represented by the second prediction error correction symbol is a text deletion operation or a text insertion operation, then the image feature vector, the preset mask character, the predicted text region of the image to be recognized, and the predicted text in the predicted text region are concatenated to obtain a new concatenation result. The above steps of performing text recognition on the concatenation result through the decoding module up to concatenating the image feature vector, the preset mask character, the predicted text region of the image to be recognized, and the predicted text in the predicted text region are repeated until the operation type represented by the obtained second prediction error correction symbol is no error correction.

10. A training device for a character recognition model, characterized in that, include: The first acquisition unit is used to acquire a training dataset, which includes sample images and corresponding real labels for the sample images. The real labels are formed by concatenating the position information of the text regions in the sample images and the real text in the text regions. The first masking unit is used to mask the real text in the real label to obtain the first masked label. The first recognition unit is configured to encode the sample image using the encoding module in the character recognition model to obtain a first image feature vector of the sample image, and then perform character recognition on the first image feature vector based on the first mask label using the decoding module in the character recognition model to obtain a first character recognition result corresponding to the sample image; the first character recognition result includes a first predicted character region of the sample image and the predicted character in the first predicted character region; the predicted character is generated by predicting and generating the masked real character based on the first image feature vector and the unmasked content in the first mask label. The first optimization unit is used to optimize the text recognition model based on the real label corresponding to the sample image and the first text recognition result.

11. A character recognition device, characterized in that, include: The second acquisition unit is used to acquire the image to be recognized; The second recognition unit is used to encode the image to be recognized through the encoding module in the character recognition model to obtain the image feature vector of the image to be recognized, and to perform character recognition on the image feature vector through the decoding module in the character recognition model to obtain the character recognition result corresponding to the image to be recognized. The character recognition model is trained based on the training method of the character recognition model according to any one of claims 1 to 7.

12. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the method as described in any one of claims 1 to 7; or the processor is configured to execute the instructions to implement the method as described in claim 8 or 9.

13. A computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is able to perform the method as described in any one of claims 1 to 7; or, when the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is able to perform the method as described in claim 8 or 9.

Images