Text recognition method and device, electronic equipment and storage medium

Abstract

The application provides a character recognition method and device, electronic equipment and storage medium, and relates to the technical field of computer vision. In the decoding process, parallel decoding, autoregressive decoding and bidirectional decoding are jointly used, the advantages of various decoding modes can be fully applied, autoregressive decoding is performed after parallel decoding, the parallel decoding result can be corrected, the accuracy of the initial decoding result is improved, and the efficiency of autoregressive decoding is improved. Through bidirectional decoding on the initial decoding result, the target recognition result is more accurate. The method can be applied to recognizing two-dimensional image structure characters or multiple lines of characters, and can also realize accurate character recognition for fuzzy or ambiguous scenes, and has strong language modeling capability.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a text recognition method, apparatus, electronic device, and storage medium. Background Technology

[0002] Text recognition technology is a technique that extracts text from images or scanned documents and outputs it in a human-readable format, such as editable text format. It is an important foundation for enabling machines to learn to read and has applications in various aspects of life, such as document recognition, license plate recognition, and business card recognition. Therefore, improving text recognition performance is of great significance to promoting the development of artificial intelligence.

[0003] Existing text recognition technologies mainly fall into two categories: The first is a decoding scheme based on Connectionist Temporal Classification (CTC). This scheme encodes images into a one-dimensional sequence and uses CTC-Loss to align labels and prediction results. The second is an autoregressive decoding scheme based on Encoder-Decoder (ED). This scheme predicts characters step-by-step in the desired order and has a wider range of applications.

[0004] However, CTC-based decoding schemes require mapping images to a one-dimensional sequence, making them unsuitable for recognizing text in two-dimensional images or multi-line text. ED-based autoregressive decoding schemes employ a unidirectional language model, where each decoding step can only utilize character information preceding the current character in the defined decoding order. This results in insufficient language modeling capabilities for ambiguous or unclear scenarios, leading to inaccurate text recognition results. Figure 1 As shown, in ambiguous scenarios, the two characters 'p' in 'Happy' might be misidentified as 'r' without reading the full text. Summary of the Invention

[0005] This invention provides a character recognition method, device, electronic device, and storage medium to address the deficiencies in related technologies.

[0006] This invention provides a character recognition method, comprising: Acquire a target image, wherein the target image includes the text to be recognized; The target image is encoded to obtain image encoding features; Based on the image encoding features, the text to be recognized is sequentially subjected to parallel decoding and autoregressive decoding to obtain an initial decoding result. Based on the image encoding features, the initial decoding result is bidirectionally decoded to obtain the target recognition result of the text to be recognized.

[0007] According to a text recognition method provided by the present invention, the step of performing parallel decoding and autoregressive decoding on the text to be recognized based on the image encoding features to obtain an initial decoding result includes: Based on the image encoding features, the text to be recognized is decoded in parallel to obtain the parallel decoding result; Based on the image encoding features, using the parallel decoding results as real data, a teacher-forced strategy is applied to perform autoregressive decoding on the text to be recognized, and the initial decoding results are obtained.

[0008] According to a text recognition method provided by the present invention, the step of performing parallel decoding and autoregressive decoding on the text to be recognized based on the image encoding features to obtain an initial decoding result further includes: The number of characters decoded in a single operation is determined, and based on the number of characters decoded in a single operation, the text to be recognized is divided into blocks to obtain multiple character blocks; Based on the image encoding features, parallel decoding is performed within each character block, and autoregressive decoding is performed between each character block to obtain the initial decoding result.

[0009] According to a character recognition method provided by the present invention, the method involves performing parallel decoding within each character block and autoregressive decoding between each character block based on the image encoding features to obtain the initial decoding result, including: For the current character block in each of the character blocks, based on the image encoding features and the autoregressive decoding result of the previous character block, the current character block is decoded in parallel to obtain the current parallel decoding result; Based on the image encoding features, using the current parallel decoding result as the real data, and applying the teacher-forced strategy, autoregressive decoding is performed on the current character block to obtain the autoregressive decoding result of the current character block; The initial decoding result is determined based on the autoregressive decoding results of each character block.

[0010] According to a text recognition method provided by the present invention, the step of bidirectionally decoding the initial decoding result based on the image encoding features to obtain the target recognition result of the text to be recognized includes: For the current character in the text to be identified, based on the image encoding features, the initial decoding results of other characters in the text to be identified are applied to decode the current character to obtain the bidirectional decoding result of the current character; The target recognition result is determined based on the bidirectional decoding results of each character in the text to be recognized.

[0011] According to a text recognition method provided by the present invention, the step of performing bidirectional decoding on the initial decoding result based on the image encoding features to obtain the target recognition result of the text to be recognized further includes: Based on the image encoding features, the initial decoding result is subjected to multiple iterative bidirectional decodings until the number of iterations of the bidirectional decoding reaches a preset number, or until the bidirectional decoding results obtained by two consecutive bidirectional decodings of each character in the text to be identified are the same. Then, based on the bidirectional decoding results of each character in the text to be identified at the current time, the target recognition result is determined.

[0012] According to a text recognition method provided by the present invention, the text to be recognized is sequentially subjected to parallel decoding and autoregressive decoding based on the image encoding features to obtain an initial decoding result, and bidirectional decoding is performed on the initial decoding result based on the image encoding features to obtain the target recognition result of the text to be recognized, which is implemented based on a text decoder; the text decoder is trained based on the following steps: Multiple image samples containing known text are acquired, and each image sample is divided into multiple training batches. Each image sample in each training batch is further divided into multiple parts. Each part of the image sample corresponds to a different decoding method, and the different decoding methods include at least autoregressive decoding, bidirectional decoding, parallel decoding, and block parallel decoding. The image encoding feature samples of each image sample are input into the initial decoder, and the initial decoder decodes the known text in each image sample according to the corresponding decoding method to obtain the decoding result; Based on the decoding results and the known text in each of the image samples, the initial decoder is iteratively trained to obtain the text decoder.

[0013] The present invention also provides a character recognition device, comprising: Image acquisition module, used to acquire target image, wherein the target image includes text to be recognized; An image encoding module is used to encode the target image to obtain image encoding features; The text decoding module is used to perform parallel decoding and autoregressive decoding on the text to be recognized based on the image encoding features to obtain an initial decoding result, and to perform bidirectional decoding on the initial decoding result based on the image encoding features to obtain the target recognition result of the text to be recognized.

[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the character recognition method as described above.

[0015] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the character recognition method as described above.

[0016] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the character recognition method as described above.

[0017] The text recognition method, apparatus, electronic device, and storage medium provided by this invention employ parallel decoding, autoregressive decoding, and bidirectional decoding in combination during the decoding process. This fully utilizes the advantages of various decoding methods. Performing autoregressive decoding after parallel decoding not only corrects the parallel decoding results and improves the accuracy of the initial decoding results but also enhances the efficiency of autoregressive decoding. By performing bidirectional decoding on the initial decoding results, the target recognition results can be made more accurate. This method is applicable not only to recognizing text in two-dimensional image structures or multi-line text but also to accurately recognizing text in ambiguous or unclear scenarios, demonstrating strong language modeling capabilities. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this invention or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of an ambiguous scenario.

[0020] Figure 2 This is a flowchart illustrating the text recognition method provided by the present invention.

[0021] Figure 3 This is a schematic diagram of the autoregressive decoding process in the text recognition method provided by the present invention.

[0022] Figure 4 This is a schematic diagram of the attention mask matrix structure used in the autoregressive decoding process of the character recognition method provided by this invention.

[0023] Figure 5 This is a schematic diagram of the bidirectional decoding process in the text recognition method provided by the present invention.

[0024] Figure 6 This is a schematic diagram of the attention mask matrix structure used in the bidirectional decoding process of the text recognition method provided by this invention.

[0025] Figure 7This is a schematic diagram of the parallel decoding process in the text recognition method provided by the present invention.

[0026] Figure 8 This is a schematic diagram of the attention mask matrix structure used in the parallel decoding process of the character recognition method provided by the present invention.

[0027] Figure 9 This is a schematic diagram of the block-parallel decoding process in the text recognition method provided by the present invention.

[0028] Figure 10 This is a schematic diagram of the attention mask matrix structure used in the block-parallel decoding process of the text recognition method provided by the present invention.

[0029] Figure 11 This is a schematic diagram of the character recognition device provided by the present invention.

[0030] Figure 12 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0032] Because existing character recognition technologies, such as CTC-based decoding schemes, are not suitable for recognizing characters in two-dimensional image structures or multi-line text, and ED-based autoregressive decoding schemes lack sufficient language modeling capabilities for ambiguous or unclear scenarios, the character recognition results are inaccurate.

[0033] Based on this, an embodiment of the present invention provides a text recognition method. For example... Figure 2 As shown, the method includes: S1, acquire a target image, wherein the target image includes text to be recognized; S2, Encode the target image to obtain image coding features; S3. Based on the image encoding features, the text to be recognized is sequentially subjected to parallel decoding and autoregressive decoding to obtain an initial decoding result. Based on the image encoding features, the initial decoding result is subjected to bidirectional decoding to obtain the target recognition result of the text to be recognized.

[0034] Specifically, the text recognition method provided in this embodiment of the invention is executed by a text recognition device, which can be configured in a computer. The computer can be a local computer or a cloud computer. The local computer can be a computer, tablet, etc., and no specific limitation is made here.

[0035] First, step S1 is executed to acquire a target image, which includes the text to be recognized. If this method is applied to document recognition, the target image can be a document image, and the text to be recognized is the document's text content; if this method is applied to license plate recognition, the target image can be a license plate image, and the text to be recognized is the license plate number; if this method is applied to business card recognition, the target image can be a business card image, and the text to be recognized is the business card's text content; if this method is applied to ship license plate recognition, the target image can be a ship license plate image, and the text to be recognized is the ship license plate number.

[0036] The target image can be acquired using an image acquisition device, such as a camera or webcam.

[0037] Then, step S2 is executed, where the target image is encoded using an image encoder to obtain image encoded features. The image encoder can be a convolutional neural network (CNN), such as ResNet or SwingTransformer.

[0038] Finally, step S3 is executed, which utilizes image encoding features and applies an attention mechanism to sequentially perform parallel decoding and autoregressive decoding on the text to be recognized, thereby obtaining the initial decoding result.

[0039] Parallel decoding is a process that utilizes image-encoded features and applies an attention mechanism to decode each character in the text to be recognized in parallel. The inputs to the parallel decoding process include image-encoded features and the start marker of the text to be recognized. <s>The output is the parallel encoding result of each character in the text to be recognized. Here, the number of starting markers can be the same as the number of characters in the text to be recognized. The remaining inputs to the parallel decoding process, excluding the image encoding features, can be controlled by an attention mask matrix.

[0040] Autoregressive decoding is a process that utilizes the parallel decoding results of each character in the text to be recognized and applies an attention mechanism to perform autoregressive decoding on each character. Here, the input to the current regression step in the autoregressive decoding process includes image-encoded features, the input of the initial regression step, and the parallel decoding results of the characters corresponding to all regression steps before the current regression step. The remaining inputs in the autoregressive decoding process, excluding image-encoded features, can be controlled using an attention mask matrix.

[0041] After the autoregressive decoding process is completed, the autoregressive decoding results of each character can be used as the initial decoding results of each character.

[0042] Subsequently, by utilizing image coding features, the initial decoding results of each character can be bidirectionally decoded, making full use of the rich contextual information of each character to obtain the target recognition results of each character in the text to be recognized.

[0043] As can be understood, bidirectional decoding refers to the process of decoding the current character using image-encoded features and the initial decoding results of characters other than the current character. The inputs to the bidirectional decoding process include image-encoded features and the initial decoding results of characters other than the current character. The remaining inputs to the bidirectional decoding process, excluding image-encoded features, can be controlled using an attention mask matrix.

[0044] The text recognition method provided in this embodiment of the invention first acquires a target image containing the text to be recognized. Then, the target image is encoded to obtain image encoding features. Finally, based on the image encoding features, the text to be recognized is sequentially subjected to parallel decoding and autoregressive decoding to obtain an initial decoding result. Based on the image encoding features, the initial decoding result is then subjected to bidirectional decoding to obtain the target recognition result of the text to be recognized. This method combines parallel decoding, autoregressive decoding, and bidirectional decoding during the decoding process, fully utilizing the advantages of various decoding methods. Performing autoregressive decoding after parallel decoding not only corrects the parallel decoding result, improving the accuracy of the initial decoding result, but also improves the efficiency of autoregressive decoding. By performing bidirectional decoding on the initial decoding result, the target recognition result can be made more accurate. This method is not only applicable to recognizing text in two-dimensional image structures or multi-line text, but also achieves accurate text recognition in ambiguous or unclear scenarios, demonstrating strong language modeling capabilities.

[0045] Based on the above embodiments, the step of performing parallel decoding and autoregressive decoding on the text to be recognized based on the image encoding features to obtain an initial decoding result includes: Based on the image encoding features, the text to be recognized is decoded in parallel to obtain the parallel decoding result; Based on the image encoding features, using the parallel decoding results as real data, a teacher-forced strategy is applied to perform autoregressive decoding on the text to be recognized, and the initial decoding results are obtained.

[0046] Specifically, in the process of performing parallel decoding and autoregressive decoding on the text to be recognized, image coding features can be used first, and an attention mechanism can be applied to perform parallel decoding on the text to be recognized, so as to obtain the parallel decoding result.

[0047] Then, using image encoding features and the parallel decoding results as real data, a teacher forcing strategy is applied to perform autoregressive decoding on the text to be recognized, yielding an initial decoding result. The teacher forcing strategy refers to a strategy in which the current regression step's autoregressive decoding process does not rely on the autoregressive decoding result of the previous regression step, but rather uses the real data as the teacher signal for autoregressive decoding. In other words, the input to the current regression step's autoregressive decoding process does not include the autoregressive decoding result of the previous regression step.

[0048] It is understood that the input to the existing autoregressive decoding process includes image encoded features, the input of the initial regression step, and the autoregressive decoding results corresponding to all regression steps before the current regression step. In contrast, the embodiments of the present invention introduce Teacher Forcing, using parallel decoding results as real data to replace the autoregressive decoding results in the input. This not only provides higher quality input data for the autoregressive decoding process but also shortens the autoregressive decoding process and improves its efficiency.

[0049] Based on the above embodiments, the step of performing parallel decoding and autoregressive decoding on the text to be recognized sequentially based on the image encoding features to obtain an initial decoding result further includes: The number of characters decoded in a single operation is determined, and based on the number of characters decoded in a single operation, the text to be recognized is divided into blocks to obtain multiple character blocks; Based on the image encoding features, parallel decoding is performed within each character block, and autoregressive decoding is performed between each character block to obtain the initial decoding result.

[0050] Specifically, to further improve decoding efficiency, when performing parallel decoding and autoregressive decoding on the text to be recognized sequentially, a method of alternating block-based parallel decoding and autoregressive decoding can be adopted.

[0051] Here, block-based parallel decoding refers to the process of dividing the text to be recognized into blocks, utilizing image encoding features, and combining an attention mechanism to iteratively perform parallel decoding and autoregressive decoding on each character block. Specifically, parallel decoding is performed within each character block, while autoregressive decoding is performed between character blocks. The input to the current regression step in the block-based parallel decoding process includes image encoding features, the start marker corresponding to the character block in the current regression step, and the autoregressive decoding results output by all previous regression steps. The number of start markers corresponding to the character block in the current regression step is equal to the number of characters in each character block.

[0052] When dividing the text to be recognized into blocks, the number of characters to be decoded in a single operation can be determined first. The number of characters to be decoded in a single operation can be set as needed, and no specific limit is set here.

[0053] Furthermore, by utilizing the number of characters decoded in a single operation, the text to be recognized can be divided into blocks, resulting in multiple character blocks.

[0054] In this embodiment of the invention, by introducing block-based parallel decoding, not only can the problem of decreased prediction accuracy of decoding length and decoded characters due to excessively long character sequences of the text to be recognized be solved, thus improving the accuracy of the initial decoding result, but the reasoning speed of the target recognition result can also be improved. For example, if the text to be recognized in the target image contains 100 characters, a conventional autoregressive decoding process would require 100 decoding steps. If the number of characters to be decoded per word is set to 10, then according to the text recognition method provided in this embodiment of the invention, the parallel decoding and autoregressive decoding processes for each character block are iteratively performed. Both the parallel decoding process and the autoregressive decoding process require 100 / 10 = 10 decoding steps, while the bidirectional decoding process only requires one. Therefore, for 100 characters of the text to be recognized, only 2*(100 / 10) + 1 = 21 decoding steps are needed.

[0055] Based on the above embodiments, the step of performing parallel decoding within each character block and autoregressive decoding between each character block based on the image encoding features to obtain the initial decoding result includes: For the current character block in each of the character blocks, based on the image encoding features and the autoregressive decoding result of the previous character block, the current character block is decoded in parallel to obtain the current parallel decoding result; Based on the image encoding features, using the current parallel decoding result as the real data, and applying the teacher-forced strategy, autoregressive decoding is performed on the current character block to obtain the autoregressive decoding result of the current character block; The initial decoding result is determined based on the autoregressive decoding results of each character block.

[0056] Specifically, during the parallel decoding process within each character block and the autoregressive decoding process between character blocks, for the current character block within each block, image encoding features and the autoregressive decoding result of the previous character block can be used, combined with an attention mechanism, to perform parallel decoding on the current character block and obtain the current parallel decoding result. It can be understood that this process is a parallel decoding process performed within the current character block.

[0057] Specifically, if the current character block is the first character block, then there is no previous character block. Only the image encoding features are used, combined with the attention mechanism, to perform parallel decoding on the current character block and obtain the current parallel decoding result. That is, the image encoding features and the start markers corresponding to each character in the current character block are used as input to perform parallel decoding on the current character block and obtain the current parallel decoding result.

[0058] Subsequently, utilizing image encoding features and using the current parallel decoding result as the real data, a teacher-mandated strategy combined with an attention mechanism is applied to perform autoregressive decoding on the current character block, yielding the autoregressive decoding result for the current character block. It can be understood that this process involves autoregressive decoding between the current character block and the next character block.

[0059] Finally, the autoregressive decoding result of each character in each character block can be used as the initial decoding result of each character.

[0060] In this embodiment of the invention, by introducing a teacher-forced strategy to use the current parallel decoding result in the autoregressive decoding process of the current character block, the autoregressive decoding process of the current character block can be made more efficient and the autoregressive decoding result more accurate.

[0061] Based on the above embodiments, the step of performing bidirectional decoding on the initial decoding result based on the image encoding features to obtain the target recognition result of the text to be recognized includes: For the current character in the text to be identified, based on the image encoding features, the initial decoding results of other characters in the text to be identified are applied to decode the current character to obtain the bidirectional decoding result of the current character; The target recognition result is determined based on the bidirectional decoding results of each character in the text to be recognized.

[0062] Specifically, during the bidirectional decoding process of the initial decoding result, for the current character in the text to be recognized, image encoding features are used, along with the initial decoding results of other characters in the text excluding those to be recognized, and an attention mechanism is employed to decode the current character, resulting in the bidirectional decoding result of the current character. It can be understood that the bidirectional decoding result of the current character can be obtained by correcting the initial decoding result of the current character.

[0063] After that, the bidirectional decoding results of each character in the text to be recognized can be directly used as the target recognition results of each character.

[0064] In this embodiment of the invention, bidirectional decoding can be used to correct the initial decoding result, thereby ensuring the accuracy of the target recognition result.

[0065] Based on the above embodiments, the step of performing bidirectional decoding on the initial decoding result based on the image encoding features to obtain the target recognition result of the text to be recognized further includes: Based on the image encoding features, the initial decoding result is subjected to multiple iterative bidirectional decodings until the number of iterations of the bidirectional decoding reaches a preset number, or until the bidirectional decoding results obtained by two consecutive bidirectional decodings of each character in the text to be identified are the same. Then, based on the bidirectional decoding results of each character in the text to be identified at the current time, the target recognition result is determined.

[0066] Specifically, to further improve the accuracy of target recognition results, during the bidirectional decoding process of the initial decoding result, image coding features can be utilized and an attention mechanism applied to perform multiple iterative bidirectional decodings on the initial decoding result. That is, after obtaining the bidirectional decoding results of each character in the text to be recognized, image coding features are used again and an attention mechanism is applied to perform bidirectional decoding on the bidirectional decoding results of each character. Each bidirectional decoding result is a result obtained by correcting the input decoding result.

[0067] The bidirectional decoding ends when the preset number of iterations is reached, or when the bidirectional decoding results of two consecutive bidirectional decoding operations on each character in the text to be recognized are identical. In this case, the bidirectional decoding result of each character in the text at the current moment can be used as the target recognition result for each character. The preset number of iterations can be set as needed, for example, to 3.

[0068] In this embodiment of the invention, by iterating bidirectional decoding multiple times, the effect of bidirectional decoding can be further guaranteed, and the accuracy of target recognition results for each character can be improved.

[0069] Based on the above embodiments, the text to be recognized is sequentially subjected to parallel decoding and autoregressive decoding based on the image encoding features to obtain an initial decoding result. Then, based on the image encoding features, the initial decoding result is bidirectionally decoded to obtain the target recognition result of the text to be recognized. This is implemented based on a text decoder. The text decoder is trained based on the following steps: Multiple image samples containing known text are acquired, and each image sample is divided into multiple training batches. Each image sample in each training batch is further divided into multiple parts. Each part of the image sample corresponds to a different decoding method, and the different decoding methods include at least autoregressive decoding, bidirectional decoding, parallel decoding, and block parallel decoding. The image encoding feature samples of each image sample are input into the initial decoder, and the initial decoder decodes the known text in each image sample according to the corresponding decoding method to obtain the decoding result; Based on the decoding results and the known text in each of the image samples, the initial decoder is iteratively trained to obtain the text decoder.

[0070] Specifically, each decoding mode in the embodiments of the present invention is implemented through a text decoder, which can be a Transformer network. The attention mask matrix in the Transformer network allows for flexible control of autoregressive decoding, bidirectional decoding, parallel decoding, and block-parallel decoding processes.

[0071] A text decoder can be trained by using multiple image samples with known text and different decoding methods to train an initial decoder, so that the trained text decoder can support different decoding modes.

[0072] During the training of the initial decoder, the image samples can be divided into multiple training batches, and each training batch can be further divided into multiple parts; each part of the image samples corresponds to a different decoding method. The decoding methods include at least autoregressive decoding, bidirectional decoding, parallel decoding, and block parallel decoding, and other decoding methods may also be included, which are not specifically limited here.

[0073] To ensure that each decoding method has the same training effect, each image sample in each training batch can be divided into multiple parts. For example, if the decoding methods include autoregressive decoding, bidirectional decoding, parallel decoding, and block parallel decoding, then 1 / 4 of the image samples in each training batch correspond to autoregressive decoding, 1 / 4 correspond to bidirectional decoding, 1 / 4 correspond to parallel decoding, and 1 / 4 correspond to block parallel decoding.

[0074] Taking the known text "The weather is nice today" in an image sample as an example, this paper explains the autoregressive decoding process, bidirectional decoding process, parallel decoding process, block-parallel decoding process, and their attention mask matrices. The autoregressive decoding process is as follows: Figure 3 As shown, Figure 3 The bottom line shows the input for each regression step. The input for the current regression step is the input of the initial regression step and the decoding results obtained from all previous regression steps. For example, the input for the initial regression step is the start marker of the known text, represented as... <s>If the output of the initial regression step is "now", then the input of the second regression step includes... <s>The output of the second regression step is "day", and the input of the third regression step includes "today". <s>,"Jin" and "Tian", and so on. Figure 3 Only a part of the corresponding relations are given herein. For detailed corresponding relations, please refer to Figure 4 .

[0075] The attention mask matrix adopted in the autoregressive decoding process can be as shown in Figure 4 , Figure 4 rows represent inputs of the autoregressive decoding process, and columns represent outputs of the autoregressive decoding process. Figure 4 In each row, gray indicates that the input of the corresponding column can be attended to in attention calculation, and white indicates that the input of the corresponding column cannot be attended to in attention calculation.

[0076] The bidirectional decoding process is as shown in Figure 5 , Figure 5 the lower row is an input of the bidirectional decoding process, and the upper row is an output of the bidirectional decoding process, and both the input and the output of the bidirectional decoding process can be each character in known text. Figure 5 Only a part of the corresponding relations are given herein. For detailed corresponding relations, please refer to Figure 6 . The attention mask matrix adopted in the bidirectional decoding process can be as shown in Figure 6 , Figure 6 rows represent inputs of the bidirectional decoding process, and columns represent outputs of the bidirectional decoding process. Figure 6 In each row, gray indicates that the input of the corresponding column can be attended to in attention calculation, and white indicates that the input of the corresponding column cannot be attended to in attention calculation. From Figure 6 it can be seen that in the bidirectional decoding process, when each character is subjected to bidirectional decoding, each character itself does not participate in attention calculation, and only other characters participate in attention calculation.

[0077] The parallel decoding process is as shown in Figure 7 , in Figure 7 , the lower row is an input of the parallel decoding process, and the upper row is an output of the parallel decoding process, <s>All of these are starting marks for known text, and their number is the same as the number of each character in the known text. Figure 7 Only a partial correspondence is given in the text; for a detailed correspondence, please refer to [link to relevant documentation]. Figure 8 The attention mask matrix used in the parallel decoding process can be as follows: Figure 8 As shown, Figure 8 The rows represent the inputs to the parallel decoding process, and the columns represent the outputs of the parallel decoding process. Figure 8 In each row, gray indicates that the input in the corresponding column can be focused on during attention calculation, while white indicates that the input in the corresponding column cannot be focused on during attention calculation. Figure 8 As can be seen from this, during the parallel decoding process, when decoding each character in parallel, the starting marker corresponding to each character participates in the attention calculation.

[0078] The block-based parallel decoding process is as follows: Figure 9 As shown, in Figure 9 In the diagram, each line below the connecting line represents the input for each regression step in the block decoding process, and the line above the connecting line represents the output of the block decoding process. Each line below the connecting line contains... <s>Each character is the start marker corresponding to a character block, and each line contains... <s>The number is the same as the number of each character in the corresponding character block. Figure 9 The example shown here is that the number of characters decoded in a single operation is 2, meaning that each character block contains 2 characters. Figure 9 Only a partial correspondence is given in the text; for a detailed correspondence, please refer to [link to relevant documentation]. Figure 10 .

[0079] The attention mask matrix used in the block-parallel decoding process can be as follows: Figure 10 As shown, Figure 10 It consists of three sub-matrices, each corresponding to a character block. The rows of the sub-matrix corresponding to each character block represent the input of the parallel decoding process for that character block, and the columns represent the output of the decoding process for that character block. Figure 10 In each row, gray indicates that the input in the corresponding column can be focused on during attention calculation, while white indicates that the input in the corresponding column cannot be focused on during attention calculation. Figure 10 As can be seen, during the block-based parallel decoding process, when decoding each character in each character block in parallel, the start marker corresponding to each character and the parallel decoding result of the previous character block are both involved in the attention calculation.

[0080] Subsequently, the image encoding feature samples of each image sample are input into the initial decoder, which then decodes the known text in each image sample according to the corresponding decoding method to obtain the decoding result.

[0081] Using the decoding results and the known text in each image sample, a supervised training loss is calculated. This supervised training loss is then used to iteratively train the initial decoder. The text decoder is obtained when a specified number of iterations is reached or the supervised training loss converges. The specified number of iterations can be set as needed and is not specifically limited here.

[0082] In this embodiment of the invention, by introducing different decoding methods in each training batch during the training process of the text decoder, the text decoder can be made capable of supporting different decoding methods.

[0083] like Figure 11 As shown, based on the above embodiments, this embodiment of the invention provides a character recognition device, including: Image acquisition module 11 is used to acquire a target image, wherein the target image includes text to be recognized; Image encoding module 12 is used to encode the target image to obtain image encoding features; The text decoding module 13 is used to perform parallel decoding and autoregressive decoding on the text to be recognized based on the image encoding features to obtain an initial decoding result, and to perform bidirectional decoding on the initial decoding result based on the image encoding features to obtain the target recognition result of the text to be recognized.

[0084] Based on the above embodiments, the text recognition device provided in this embodiment of the invention, wherein the text decoding module is specifically used for: Based on the image encoding features, the text to be recognized is decoded in parallel to obtain the parallel decoding result; Based on the image encoding features, using the parallel decoding results as real data, a teacher-forced strategy is applied to perform autoregressive decoding on the text to be recognized, and the initial decoding results are obtained.

[0085] Based on the above embodiments, the text recognition device provided in this embodiment of the invention further includes a text decoding module specifically used for: The number of characters decoded in a single operation is determined, and based on the number of characters decoded in a single operation, the text to be recognized is divided into blocks to obtain multiple character blocks; Based on the image encoding features, parallel decoding is performed within each character block, and autoregressive decoding is performed between each character block to obtain the initial decoding result.

[0086] Based on the above embodiments, the text recognition device provided in this embodiment of the invention, wherein the text decoding module is specifically used for: For the current character block in each of the character blocks, based on the image encoding features and the autoregressive decoding result of the previous character block, the current character block is decoded in parallel to obtain the current parallel decoding result; Based on the image encoding features, using the current parallel decoding result as the real data, and applying the teacher-forced strategy, autoregressive decoding is performed on the current character block to obtain the autoregressive decoding result of the current character block; The initial decoding result is determined based on the autoregressive decoding results of each character block.

[0087] Based on the above embodiments, the text recognition device provided in this embodiment of the invention further includes a text decoding module specifically used for: For the current character in the text to be identified, based on the image encoding features, the initial decoding results of other characters in the text to be identified are applied to decode the current character to obtain the bidirectional decoding result of the current character; The target recognition result is determined based on the bidirectional decoding results of each character in the text to be recognized.

[0088] Based on the above embodiments, the text recognition device provided in this embodiment of the invention further includes a text decoding module specifically used for: Based on the image encoding features, the initial decoding result is subjected to multiple iterative bidirectional decodings until the number of iterations of the bidirectional decoding reaches a preset number, or until the bidirectional decoding results obtained by two consecutive bidirectional decodings of each character in the text to be identified are the same. Then, based on the bidirectional decoding results of each character in the text to be identified at the current time, the target recognition result is determined.

[0089] Based on the above embodiments, the text recognition device provided in this embodiment of the invention further includes a text decoding module specifically used for: determining the target recognition result of the text to be recognized based on the text decoder; and a training module used for: Multiple image samples containing known text are acquired, and each image sample is divided into multiple training batches. Each image sample in each training batch is further divided into four parts. The four parts of the image samples correspond to different decoding methods, including autoregressive decoding, bidirectional decoding, parallel decoding, and block parallel decoding. The image encoding feature samples of each image sample are input into the initial decoder, and the initial decoder decodes the known text in each image sample according to the corresponding decoding method to obtain the decoding result; Based on the decoding results and the known text in each of the image samples, the initial decoder is iteratively trained to obtain the text decoder.

[0090] Specifically, the functions of each module in the text recognition device provided in this embodiment of the invention correspond one-to-one with the operation flow of each step in the above method-like embodiments, and the achieved effects are also the same. For details, please refer to the above embodiments, and this will not be repeated in this embodiment of the invention.

[0091] Figure 12 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 12 As shown, the electronic device may include a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute the character recognition methods provided in the above embodiments.

[0092] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to related technologies, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0093] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the character recognition method provided in the above embodiments.

[0094] In another aspect, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the character recognition methods provided in the above embodiments. This computer-readable storage medium can be either a non-transitory computer-readable storage medium or a transient computer-readable storage medium, and is not specifically limited herein.

[0095] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0096] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0097] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.< / s> < / s> < / s> < / s> < / s> < / s> < / s>

Claims

1. A character recognition method, characterized in that, include: Acquire a target image, wherein the target image includes the text to be recognized; The target image is encoded to obtain image encoding features; Based on the image encoding features, the text to be identified is sequentially subjected to parallel decoding and autoregressive decoding to obtain an initial decoding result. Based on the image encoding features, the initial decoding result is then subjected to bidirectional decoding to obtain the target recognition result of the text to be identified. The process of performing parallel decoding and autoregressive decoding on the text to be recognized based on the image encoding features to obtain an initial decoding result includes: Based on the image encoding features, the text to be recognized is decoded in parallel to obtain the parallel decoding result; Based on the image encoding features, using the parallel decoding results as real data, a teacher-forced strategy is applied to perform autoregressive decoding on the text to be recognized, and the initial decoding results are obtained.

2. The character recognition method according to claim 1, characterized in that, The step of performing parallel decoding and autoregressive decoding on the text to be recognized based on the image encoding features to obtain an initial decoding result also includes: The number of characters decoded in a single operation is determined, and based on the number of characters decoded in a single operation, the text to be recognized is divided into blocks to obtain multiple character blocks; Based on the image encoding features, parallel decoding is performed within each character block, and autoregressive decoding is performed between each character block to obtain the initial decoding result.

3. The character recognition method according to claim 2, characterized in that, The process of performing parallel decoding within each character block and autoregressive decoding between each character block based on the image encoding features to obtain the initial decoding result includes: For the current character block in each of the character blocks, based on the image encoding features and the autoregressive decoding result of the previous character block, the current character block is decoded in parallel to obtain the current parallel decoding result; Based on the image encoding features, using the current parallel decoding result as the real data, and applying the teacher-forced strategy, autoregressive decoding is performed on the current character block to obtain the autoregressive decoding result of the current character block. The initial decoding result is determined based on the autoregressive decoding results of each character block.

4. The character recognition method according to any one of claims 1-3, characterized in that, The step of bidirectionally decoding the initial decoding result based on the image encoding features to obtain the target recognition result of the text to be recognized includes: For the current character in the text to be identified, based on the image encoding features, the initial decoding results of other characters in the text to be identified are applied to decode the current character to obtain the bidirectional decoding result of the current character; The target recognition result is determined based on the bidirectional decoding results of each character in the text to be recognized.

5. The character recognition method according to claim 4, characterized in that, The step of performing bidirectional decoding on the initial decoding result based on the image encoding features to obtain the target recognition result of the text to be recognized further includes: Based on the image encoding features, the initial decoding result is subjected to multiple iterative bidirectional decodings until the number of iterations of the bidirectional decoding reaches a preset number, or until the bidirectional decoding results obtained by two consecutive bidirectional decodings of each character in the text to be identified are the same. Then, based on the bidirectional decoding results of each character in the text to be identified at the current time, the target recognition result is determined.

6. The character recognition method according to any one of claims 1-3, characterized in that, Based on the image encoding features, the text to be recognized is sequentially subjected to parallel decoding and autoregressive decoding to obtain an initial decoding result. Then, based on the image encoding features, the initial decoding result is bidirectionally decoded to obtain the target recognition result of the text to be recognized. This is implemented based on a text decoder; the text decoder is trained based on the following steps: Multiple image samples containing known text are acquired, and each image sample is divided into multiple training batches. Each image sample in each training batch is further divided into multiple parts. Each part of the image sample corresponds to a different decoding method, and the different decoding methods include at least autoregressive decoding, bidirectional decoding, parallel decoding, and block parallel decoding. The image encoding feature samples of each image sample are input into the initial decoder, and the initial decoder decodes the known text in each image sample according to the corresponding decoding method to obtain the decoding result; Based on the decoding results and the known text in each of the image samples, the initial decoder is iteratively trained to obtain the text decoder.

7. A character recognition device, characterized in that, include: An image acquisition module is used to acquire a target image, wherein the target image includes text to be recognized; An image encoding module is used to encode the target image to obtain image encoding features; The text decoding module is used to perform parallel decoding and autoregressive decoding on the text to be recognized sequentially based on the image encoding features to obtain an initial decoding result, and to perform bidirectional decoding on the initial decoding result based on the image encoding features to obtain the target recognition result of the text to be recognized; The process of performing parallel decoding and autoregressive decoding on the text to be recognized based on the image encoding features to obtain an initial decoding result includes: Based on the image encoding features, the text to be recognized is decoded in parallel to obtain the parallel decoding result; Based on the image encoding features, using the parallel decoding results as real data, a teacher-forced strategy is applied to perform autoregressive decoding on the text to be recognized, and the initial decoding results are obtained.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the character recognition method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the character recognition method as described in any one of claims 1-6.

Images