Character recognition method, apparatus, device, and medium

Abstract

The present disclosure provides a character recognition method, which relates to the field of artificial intelligence. The method comprises: acquiring a to-be-recognized image, wherein the to-be-recognized image comprises N to-be-recognized characters arranged irregularly, the irregular arrangement comprises not being arranged in a straight line, and N is an integer greater than or equal to 2; inputting the to-be-recognized image into a character recognition model; and obtaining recognition results of the N to-be-recognized characters output by the character recognition model; wherein the character recognition model comprises an encoder and a decoder, and the character recognition model is configured to be pre-trained by: obtaining a first loss function value based on an encoding result of the encoder, obtaining a second loss function value based on a decoding result of the decoder, and updating the encoder and the decoder based on the first loss function value and the second loss function value. The present disclosure also provides a character recognition device, equipment, medium and program product.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence, and more specifically, to a character recognition method, apparatus, device, medium, and program product. Background Technology

[0002] Traditional OCR (optical character recognition) models are mainly effective for recognizing characters arranged in straight lines, such as horizontal or vertical lines. They are not very effective for recognizing characters that are irregularly arranged, such as curved lines, multi-line text, or symbols, like seals or formulas.

[0003] In related technologies, some recognition schemes for irregularly arranged characters require transforming the specially arranged text into a horizontal arrangement before using a traditional OCR recognition model for recognition. This involves cumbersome steps and has unsatisfactory accuracy and robustness. Summary of the Invention

[0004] In view of the above problems, this disclosure provides a character recognition method, apparatus, device, medium and program product.

[0005] One aspect of this disclosure provides a character recognition method, comprising: acquiring an image to be recognized, wherein the image to be recognized includes N characters to be recognized arranged irregularly, the irregular arrangement including not arranged in a straight line, and N being an integer greater than or equal to 2; inputting the image to be recognized into a character recognition model; obtaining recognition results of the N characters to be recognized output by the character recognition model; wherein the character recognition model includes an encoder and a decoder, and the character recognition model is configured to be pre-trained by: obtaining a first loss function value based on the encoding result of the encoder, obtaining a second loss function value based on the decoding result of the decoder, and updating the encoder and the decoder based on the first loss function value and the second loss function value.

[0006] According to an embodiment of this disclosure, obtaining the image to be recognized includes: determining a target image; using a target detection model to determine a first target region from the target image, the first target region including the N characters to be recognized; and segmenting the first target region from the target image to obtain the image to be recognized.

[0007] According to an embodiment of this disclosure, obtaining the recognition results of the N characters to be recognized output by the character recognition model includes: processing the image to be recognized using the encoder to obtain a first image feature output by the encoder; processing the first image feature using a classification function to obtain intermediate recognition results of the N characters to be recognized, wherein the classification function is used to classify the first image feature; inputting the first image feature and a first sequence text into the decoder to obtain the recognition results of the N characters to be recognized output by the decoder, wherein the first sequence text is obtained by right-shifting each character recognition result in the intermediate recognition results by M character positions, where M is an integer greater than or equal to 1.

[0008] According to an embodiment of this disclosure, the decoder includes a cross-attention layer and a classification layer. The step of inputting the first image features and the first sequence of text into the decoder to obtain the recognition results of the N characters to be recognized output by the decoder includes: processing the first image features and the first sequence of text using the cross-attention layer to obtain first attention features, wherein the cross-attention layer is configured to process data based on a cross-attention mechanism; and processing the first attention features using the classification layer to obtain the recognition results of the N characters to be recognized, wherein the classification layer is used to classify the first attention features.

[0009] According to an embodiment of this disclosure, obtaining the recognition results of the N characters to be recognized output by the character recognition model includes: processing the image to be recognized using the encoder to obtain a first image feature output by the encoder; inputting the first image feature into the decoder to obtain the recognition results of the N characters to be recognized output by the decoder.

[0010] According to embodiments of this disclosure, the encoding result includes second image features, and obtaining a first loss function value based on the encoding result of the encoder includes: inputting a training image into the encoder to obtain the second image features output by the encoder, wherein the training image includes N characters to be recognized in an irregular arrangement; obtaining a first prediction tensor based on the second image features, the first prediction tensor including character prediction information for the N characters to be recognized; and obtaining a first loss function value between the first prediction tensor and the labeled text, wherein the labeled text includes a label for each character to be recognized.

[0011] According to an embodiment of this disclosure, the decoding result includes a second prediction tensor, and obtaining a second loss function value based on the decoding result of the decoder includes: inputting the second image features into the decoder to obtain the second prediction tensor output by the decoder, the second prediction tensor including character prediction information for the N characters to be identified; and obtaining a second loss function value between the second prediction tensor and the labeled text.

[0012] According to embodiments of this disclosure, updating the encoder and the decoder based on the first loss function value and the second loss function value includes: obtaining a first weighted value based on the first loss function value and its first weight; obtaining a second weighted value based on the second loss function value and its second weight; obtaining a comprehensive loss function value based on the first weighted value and the second weighted value; and updating the encoder and the decoder based on the comprehensive loss function value to obtain a trained character recognition model.

[0013] According to embodiments of this disclosure, the decoder includes a cross-attention layer and a classification layer. The step of inputting the second image features into the decoder to obtain the second prediction tensor output by the decoder includes: inputting the second image features and a second sequence of text into the decoder, wherein the second sequence of text is obtained by right-shifting the label of each character to be identified in the labeled text by M character positions, where M is an integer greater than or equal to 1; processing the second image features and the second sequence of text using the cross-attention layer to obtain second attention features, wherein the cross-attention layer is configured to process data based on a cross-attention mechanism; and processing the second attention features using the classification layer to obtain the second prediction tensor.

[0014] According to an embodiment of this disclosure, obtaining the first prediction tensor based on the second image features includes: performing dimensionality upscaling on the second image features to obtain a third image feature; and inputting the third image feature into a classification function to obtain the first prediction tensor.

[0015] In another aspect of this disclosure, a character recognition device is provided, comprising: an image acquisition module for acquiring an image to be recognized, wherein the image to be recognized includes N characters to be recognized arranged irregularly, the irregular arrangement including not arranged in a straight line, and N being an integer greater than or equal to 2; a third input module for inputting the image to be recognized into a character recognition model; and a recognition result module for obtaining the recognition result of the N characters to be recognized output by the character recognition model; wherein the character recognition model includes an encoder and a decoder, and the character recognition model is configured to be pre-trained by the following operations:

[0016] A first loss function value is obtained based on the encoding result of the encoder, a second loss function value is obtained based on the decoding result of the decoder, and the encoder and the decoder are updated based on the first loss function value and the second loss function value.

[0017] Another aspect of this disclosure provides an electronic device, including: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the method as described above.

[0018] Another aspect of this disclosure provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the method described above.

[0019] Another aspect of this disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the method described above.

[0020] The above one or more embodiments have the following beneficial effects: During the pre-training process, the encoder's encoding result is not only input into the decoder, but also the first loss function value related to the encoding result is obtained, and the second loss function value obtained based on the decoder's decoding result participates in the training together, so as to obtain more information for learning, accelerate the training speed of the character recognition model, and improve the recognition effect. In character recognition scenarios, the image to be recognized can be directly recognized based on the trained character recognition model, which has the characteristics of high recognition efficiency and accuracy, good robustness, and simplicity and effectiveness in practical application scenarios. Attached Figure Description

[0021] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0022] Figure 1 The diagram illustrates a training method for a character recognition model according to embodiments of the present disclosure and an application scenario diagram of the character recognition method.

[0023] Figure 2 A flowchart illustrating a method for training a character recognition model according to an embodiment of the present disclosure is shown schematically.

[0024] Figures 3A-3B An example of a training image according to an embodiment of the present disclosure is illustrated schematically;

[0025] Figure 4 The diagram illustrates the training architecture of a character recognition model according to an embodiment of the present disclosure.

[0026] Figure 5 A flowchart illustrating the process of obtaining training images according to an embodiment of the present disclosure is shown schematically.

[0027] Figure 6 An example of a sample image according to an embodiment of the present disclosure is illustrated schematically;

[0028] Figure 7 A flowchart illustrating the process of obtaining a second prediction tensor according to an embodiment of the present disclosure is shown schematically;

[0029] Figure 8 A flowchart illustrating the process of obtaining the comprehensive loss function value according to an embodiment of the present disclosure is shown schematically;

[0030] Figure 9 A flowchart illustrating a character recognition method according to an embodiment of the present disclosure is shown schematically;

[0031] Figure 10 A flowchart illustrating the process of obtaining an image to be identified according to an embodiment of the present disclosure is shown schematically.

[0032] Figure 11 The flowchart illustrating the process of obtaining the recognition results of N characters to be recognized according to an embodiment of the present disclosure is shown in the illustration.

[0033] Figure 12 A flowchart illustrating the process of obtaining the recognition results of N characters to be recognized according to another embodiment of the present disclosure is shown schematically.

[0034] Figure 13 A flowchart illustrating an end-to-end special character recognition method based on YOLO and TrOCR according to an embodiment of the present disclosure is shown in the schematic diagram.

[0035] Figure 14 This schematic diagram illustrates a structural block diagram of a training apparatus for a character recognition model according to an embodiment of the present disclosure;

[0036] Figure 1 5 schematically illustrates a structural block diagram of a character recognition device according to an embodiment of the present disclosure; and

[0037] Figure 16 A block diagram of an electronic device suitable for implementing a character recognition model training method or a character recognition method according to embodiments of the present disclosure is shown schematically.

[0038] It should be noted that, for clarity, the dimensions of the overall / partial structure or the overall / partial region in the drawings used to describe the embodiments of this disclosure may be enlarged or reduced, i.e., these drawings are not drawn to actual scale. Detailed Implementation

[0039] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0040] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0041] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0042] When using expressions such as "at least one of A, B, and C", they should generally be interpreted in accordance with the meaning that is commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B, and C, etc.).

[0043] To facilitate understanding of the technical solutions of this disclosure, the terms used in some embodiments of this disclosure are defined as follows:

[0044] Yolo: A one-stage target detection algorithm characterized by high speed and high accuracy.

[0045] TrOCR: An OCR recognition method based on Vision Transformer and Transformer Decoder, characterized by high accuracy and wide application scenarios.

[0046] Tensor: refers to a multidimensional array, which is a generalization based on vectors and matrices.

[0047] Cross attention mechanism: also known as Cross Attention, is a variant of multi-head attention mechanism that can be used in sequence-to-sequence models, allowing the model to pay attention to different parts of the input at the same time when processing the input.

[0048] Classification function: A function used to map input data to different classification spaces to obtain classification results, such as softmax function, sigmoid function or CTC (Connectionist Temporal Classification) function, etc.

[0049] Figure 1 The diagram illustrates a training method for a character recognition model according to embodiments of the present disclosure and an application scenario of the character recognition method. It should be noted that... Figure 1 The examples shown are merely examples of system architectures that can be applied to the embodiments of this disclosure, in order to help those skilled in the art understand the technical content of this disclosure, but do not mean that the embodiments of this disclosure cannot be used in other devices, systems, environments or scenarios.

[0050] like Figure 1 As shown, application scenario 100 according to this embodiment may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as a medium for providing a communication link between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0051] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0052] Terminal devices 101, 102, and 103 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0053] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using terminal devices 101, 102, and 103 (for example only). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0054] Server 105 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud computing, network services, and middleware services.

[0055] In some embodiments, a training set can be prepared on terminal devices 101, 102, and 103, and server 105 can be run to train the model. Terminal devices 101, 102, and 103 can capture images of characters to be recognized using a camera or download images of characters to be recognized from the network, and send recognition requests to server 105 to obtain recognition results returned by server 105 running the character recognition model. In some embodiments, the character recognition model can be deployed on terminal devices 101, 102, and 103, and the training and recognition processes can be performed locally on terminal devices 101, 102, and 103.

[0056] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0057] The following will be based on Figure 1 The described scene, through Figures 2-8 The training method of the character recognition model according to the embodiments of this disclosure will be described in detail.

[0058] Figure 2 A flowchart illustrating a training method for a character recognition model according to an embodiment of the present disclosure is shown. Figures 3A-3B An example of a training image is illustrated schematically according to an embodiment of the present disclosure. Figure 4 A schematic diagram illustrating the training architecture of a character recognition model according to an embodiment of the present disclosure is provided. The character recognition model includes an encoder and a decoder.

[0059] like Figure 2 As shown, the training method for the character recognition model in this embodiment includes:

[0060] In operation S210, the training image is input into the encoder of the character recognition model to obtain the second image feature output by the encoder. The encoding result of the encoder includes the second image feature. The training image includes N characters to be recognized in an irregular arrangement. The irregular arrangement includes characters not arranged in a straight line. N is an integer greater than or equal to 2.

[0061] In this embodiment of the disclosure, a large number of training images need to be collected before training the model. These large number of training images may include text images of different types. Figures 3A-3B Training images can include seal images and formula images, as well as other irregularly arranged text or symbols. Linear arrangement includes writing from left to right or from top to bottom, with all characters roughly on a straight line that almost coincides with the horizontal / vertical direction.

[0062] Reference Figure 3AThe various shapes of seals can affect the arrangement of the characters, or some seal characters may be complete text, while curved text refers to characters that are almost not on the same straight line, and whose center points, when connected, roughly form a curve. Additionally, due to the different uses of seals, overlapping text may occur. Figure 3A The irregularly arranged text shown below overlaps with the regularly arranged text.

[0063] Continue to refer to Figure 3A Each seal image includes at least one seal, which can be various types of seals such as contract seals and invoice seals. The seal can be a regular shape such as a round seal, an oval seal, or a polygonal seal (e.g., a rectangular seal), or it can be an irregular shape.

[0064] Reference Figure 3B The top image shows a handwritten formula, and the bottom image shows the LaTeX language label text. Training images can also be printed formula images. For example... Figure 3B The image shown, which conveys a set of information to the reader through multi-line arrangement, is also within the scope of the irregular arrangement of this disclosure; in other words, it does not include formulaic characters arranged in multiple lines that are not in a straight line.

[0065] Training images can be text images uploaded by users or downloaded from the network, or text images captured directly by electronic devices through a camera. There are no restrictions on the source of training images.

[0066] In operation S210, the encoder is used to extract the second image features from the training images. (Refer to...) Figure 4 The illustrated TroOCR model architecture allows the encoder to be a Vision Transformer model. The Vision Transformer model is a classification model that combines Computer Vision (CV) and Natural Language Processing (NLP). It divides the original image into blocks, flattens them into sequences, inputs these sequences into the encoding part, and then processes the encoded output to complete the classification task. It is important to note that this disclosure does not intend to limit the encoder in operation S210 to a Vision Transformer; it can be flexibly replaced with other image feature extraction models, such as the Swin Transformer or CSwin Transformer.

[0067] In operation S220, a first prediction tensor is obtained based on the second image features. The first prediction tensor includes character prediction information for N characters to be identified.

[0068] For example, predictions can be made directly based on the second image features. For instance, a first prediction tensor can be obtained using a classification function, a neural network-based classification model, or a classification model based on other machine learning algorithms. The character prediction information represents the information from which the prediction results can be obtained from the dictionary.

[0069] Specifically, the number of training images is denoted as Batch_size, which can be an integer greater than or equal to 1. A number of training images equal to Batch_size are input into the encoder, and the first image feature is represented as Batch_size × L × C, where L represents the length of the character to be recognized, and C represents the dimension of the tensor.

[0070] In some embodiments, the second image features are up-dimensioned to obtain a third image feature. The third image feature is then input into a classification function to obtain a first prediction tensor. For example, a linear transformation (for example only) is performed on the first image features of batch_size×L×C to achieve up-dimension processing, resulting in batch_size×L×D, where D is the dimension of the dictionary. Then, the second image features of batch_size×L×D are input into the classification function to obtain the first prediction tensor. After up-dimensioning, the third image feature has the same dimension as the dictionary, facilitating classification.

[0071] In operation S230, the second image features are input into the decoder to obtain the second prediction tensor output by the decoder. The decoding result of the decoder includes the second prediction tensor, which includes character prediction information for N characters to be recognized.

[0072] Reference Figure 4 In operation S230, the decoder can be a Transformer Decoder, which is the decoder in a natural language processing model that performs natural language processing tasks. It can make predictions based on the first image features to obtain a second prediction tensor, and the character prediction information also represents the information that can be obtained from the dictionary.

[0073] In operation S240, a first loss function value between a first prediction tensor and labeled text, and a second loss function value between a second prediction tensor and labeled text are obtained, wherein the labeled text includes a label for each character to be identified.

[0074] The labeled text is the tag text of the character to be recognized in the training image, such as... Figure 3AThe stamp image shown is used for training, and the corresponding labeled text is "XX Co., Ltd." The ideal result of the first or second prediction tensor is the recognition of "XX Co., Ltd." The dictionary includes mappable characters for translation, representing all translatable characters used by the model. Based on the dictionary, each character can be mapped to an integer (token). For example, "XX Co., Ltd." is converted into a token integer sequence: "[1919, 25, 103, 429, 17, 151]". The first or second loss function value is calculated numerically using the token sequence of the text.

[0075] In operation S250, the comprehensive loss function value is obtained based on the first loss function value and the second loss function value.

[0076] In operation S260, the encoder and decoder are updated based on the comprehensive loss function value to obtain the trained character recognition model.

[0077] Based on the comprehensive loss function value, optimizers such as Adam or AdamW can be used to train the model until the training image input is completed or the comprehensive loss function value reaches the target convergence condition.

[0078] By obtaining the first and second prediction tensors through operations S220 and S230 as described above, the character prediction result can be directly output. Since the model can directly extract text features without rotating the text in the seal image, it avoids the problems of arbitrary seal rotation and inaccurate recognition results caused by changes in the arrangement of text on the paper.

[0079] According to embodiments of this disclosure, more information can be learned from the outputs of the encoder and decoder during training, accelerating the training speed and improving the recognition performance of the character recognition model. Recognition can be performed directly based on extracted image features, and the trained character recognition model exhibits high recognition efficiency and accuracy, good robustness, and simplicity and effectiveness in practical applications.

[0080] It should be noted that some steps of the above method can be executed individually or in combination, and can be executed in parallel or sequentially, and are not limited to the specific order of operations shown in the figure. For example, after calculating the first prediction tensor in operation S220, the first loss function value can be calculated in operation S240, without having to calculate the first loss function value after operation S230.

[0081] In related technologies, there are one-stage end-to-end scene text recognition methods such as ABCNet and PGNet. These algorithms simultaneously include text detection, transformation, and recognition functions, but the recognition accuracy is not very high, and the annotation cost is relatively high, requiring the annotation of the position of each character. If a seal on an image has 20 characters, then approximately 40 coordinate points need to be annotated, which is quite costly. This disclosure proposes a two-stage approach, where the first stage is as follows: Figure 5 The example shown first obtains the training images. Another stage is as follows: Figure 2 The example shown is for training the model.

[0082] Figure 5 A flowchart illustrating the process of obtaining training images according to an embodiment of this disclosure is shown schematically. Figure 5 As shown, this embodiment obtains training images including:

[0083] In operation S510, the sample image is determined.

[0084] For example, the sample image can be any image including a seal or formula, such as an invoice image, a receipt image, or a document image, etc., and this disclosure is not limited thereto.

[0085] Figure 6 An example of a sample image according to an embodiment of this disclosure is illustrated. Figure 6 Some information in the illustration has been occluded using texture images to avoid adverse effects. For example... Figure 6 As shown, the receipt includes a circular stamp, which is a special stamp for receipts and includes relevant text information about the stamp.

[0086] For example, the sample image can be the original image directly acquired by the image acquisition device, or it can be an image obtained after preprocessing the original image. For example, to avoid the impact of data quality or data imbalance of the input image on seal image recognition, preprocessing of the input image can be included before processing the sample image. Preprocessing can remove irrelevant information or noise information in the input image, so as to better process the input image. Preprocessing may include scaling, cropping, gamma correction, image enhancement, or noise reduction filtering of the input image.

[0087] In operation S520, a target detection model is used to determine a second target region from the sample image. The second target region includes N characters to be recognized.

[0088] In operation S530, the second target region is segmented from the sample image to obtain the training image.

[0089] According to embodiments of this disclosure, a two-stage end-to-end OCR recognition method is proposed, which performs well on text symbols with arbitrary arrangements. This method only requires labeling the coordinates and character representations of the text and symbols in the image as a whole, without needing to label the coordinates of each specially arranged text symbol, making it simple and effective.

[0090] In addition, refer to Figure 6 Sample images may include additional text such as amounts, customer signatures, or receipt headers, and may also contain multiple stamps. Determining the second target region allows for accurate input data into the character recognition model, improving character recognition accuracy.

[0091] In some embodiments, the classification function includes a connection-time classification function. Obtaining the first loss function value between the first predicted tensor and the labeled text in operation S240 includes: inputting the first predicted tensor and the labeled text into the first loss function to obtain the first loss function value, wherein the first loss function is constructed based on the connection-time classification function and is used to compare the differences between the first predicted tensor and the labeled text. (Refer to...) Figure 4 The first loss function can be CTCloss.

[0092] According to embodiments of this disclosure, the first loss function value is obtained to characterize the difference between the first predicted tensor and the labeled text, and works in conjunction with the difference between the second predicted tensor obtained by the decoder and the labeled text to participate in the training process of the character recognition model. This allows for the extraction of more image and character information for learning, thereby improving the model's convergence speed and recognition accuracy.

[0093] Figure 7 A flowchart illustrating the process of obtaining a second prediction tensor according to an embodiment of this disclosure is shown schematically. Figure 7 As shown, this embodiment is one embodiment of operation S230, and obtaining the second prediction tensor includes:

[0094] In operation S710, the second image features and the second sequence text are input into the decoder of the character recognition model. The second sequence text is obtained by right-shifting the label of each character to be recognized in the labeled text by M character positions, where M is an integer greater than or equal to 1. The decoder includes a cross-attention layer and a classification layer. In some embodiments, M may be equal to 1.

[0095] In operation S720, the second image features and the second sequence text are processed using a cross-attention layer to obtain the second attention features, wherein the cross-attention layer is configured to process data based on the cross-attention mechanism.

[0096] In the S730, the second attention feature is processed using the classification layer to obtain the second prediction tensor.

[0097] According to embodiments of this disclosure, the second prediction tensor is obtained by the decoder through a cross-attention layer and a classification layer based on the second image features and the second sequence text. Since the second sequence text is obtained by shifting the label of each character to be identified in the label text to the right by M character positions, the decoder can fully obtain contextual information during data processing, and the output recognition result is more accurate.

[0098] In some embodiments, obtaining the second loss function value between the second predicted tensor and the labeled text in operation S240 includes: inputting the second predicted tensor and the labeled text into the second loss function to obtain the second loss function value, wherein the second loss function is used to compare the difference between the second predicted tensor and the labeled text.

[0099] Reference Figure 4 The second loss function can be a classification loss function, specifically, it can be a loss function such as CrossEntropy or KLDivLoss.

[0100] Figure 8 A flowchart illustrating the process of obtaining the comprehensive loss function value according to an embodiment of this disclosure is shown schematically. Figure 8 As shown, this embodiment is one example of operating S250, and the obtained comprehensive loss function value includes:

[0101] In operation S810, the first weighted value is obtained based on the first loss function value and its first weight.

[0102] In operation S820, the second weighted value is obtained based on the second loss function value and its second weight.

[0103] During operation of S830, the comprehensive loss function value is obtained based on the first weighted value and the second weighted value.

[0104] Reference Figure 4 The third loss function can be a weighted mixed loss function, Loss, as shown in Equation 1.

[0105] Loss=αCTCLoss(y,y′1)+(1-α)ClassfyLoss(y,y′2) Equation 1

[0106] Where α is the first weight, with a value between 0 and 1. 1-α is the second weight. CTCLoss(y, y′1) is the first loss function value, ClassfyLoss(y, y′2) is the second loss function value, ClassfyLoss is the second loss function, y is the sequence vector of the labeled text, y′1 is the first prediction tensor, and y′2 is the second prediction tensor.

[0107] The first and second loss function values ​​reflect the different optimization states of the encoder and decoder, and their respective weights are used to balance the importance between different encoders and decoders. The value of α can be manually assigned a fixed value, or it can be assigned an initial value and then dynamically updated during training.

[0108] For example, when using gradient descent to update encoder and decoder parameters and α values ​​in reverse, a larger weight for a loss function value indicates a more important task (encoder or decoder) and a greater contribution to the gradient, and vice versa. The weights can be automatically adjusted during training, further influencing the parameter updates in the encoder and decoder. Tasks with larger weights have larger gradients in their loss functions, causing the model parameters to move towards the optimal solution for that task more quickly. Tasks with smaller weights have smaller gradients in their loss functions, causing the model parameters to move towards the optimal solution more slowly.

[0109] According to embodiments of this disclosure, multiple factors affecting recognition accuracy, such as second image features, first prediction tensor, and second prediction tensor, can be considered during the training phase, thereby achieving the effect of optimizing recognition accuracy during actual recognition.

[0110] After the training phase, the trained character recognition model is deployed to real-world application scenarios. (As mentioned above...) Figures 2-8 Based on this, the following combination Figures 9-12 Further explanation of the character recognition method.

[0111] Figure 9 A flowchart illustrating a character recognition method according to an embodiment of this disclosure is shown schematically. Figure 9 As shown, the character recognition method in this embodiment includes:

[0112] In operation S910, after acquiring the image to be recognized, the image to be recognized is input into the character recognition model. The image to be recognized includes N characters to be recognized in an irregular arrangement. The irregular arrangement includes characters not arranged in a straight line, and N is an integer greater than or equal to 2.

[0113] For example, the image to be identified can be as follows: Figures 3A-3B As shown. The image to be recognized can be a text image uploaded by the user or downloaded from the network, or a text image captured directly by an electronic device through a camera. There are no restrictions on the source of the image to be recognized.

[0114] The character recognition model is configured to be pre-trained through the following operations: obtaining a first loss function value based on the encoder's encoding result, obtaining a second loss function value based on the decoder's decoding result, and updating the encoder and decoder based on the first and second loss function values. Specifically, see, for example... Figures 2-8The training method described in any of the embodiments described.

[0115] By operating the S920, the recognition results of N characters to be recognized are obtained from the output of the character recognition model.

[0116] If the image to be recognized is a seal image, the recognition result may include the text of the seal, such as... Figure 3A As shown. If the image to be recognized is a formula image, the recognition result can be formula symbols or LaTeX language, such as... Figure 3B As shown on the bottom.

[0117] According to embodiments of this disclosure, text features can be extracted directly without rotating or performing other operations on the text in an image, essentially "translating" the recognition result directly from the image to be recognized. It can efficiently recognize various special character images such as seals, formulas, and multi-line text, and features high accuracy, robustness, simplicity, and effectiveness.

[0118] Figure 10 A flowchart illustrating the process of obtaining an image to be identified according to an embodiment of this disclosure is shown schematically. Figure 10 As shown, this embodiment obtains training images including:

[0119] In operation S1010, the target image is determined. The target image can be as follows: Figure 6 As shown, but this disclosure is not limited to this, it can also be document images, invoice images, etc.

[0120] In operation S1020, the first target region is determined from the target image using the target detection model. The first target region includes N characters to be recognized.

[0121] In operation S1030, the first target region is segmented from the target image to obtain the image to be recognized.

[0122] According to embodiments of this disclosure, a two-stage end-to-end OCR recognition method is used. By determining the first target region, accurate data can be input into the character recognition model, thereby improving the accuracy of character recognition. It has a good effect on any arrangement of text symbols.

[0123] Figure 11 A flowchart illustrating the process of obtaining the recognition results of N characters to be recognized according to an embodiment of this disclosure is shown. Figure 11 As shown, this embodiment is one example of operation S910, obtaining N characters to be recognized includes:

[0124] In operation S1110, the encoder of the character recognition model processes the image to be recognized and obtains the first image features output by the encoder.

[0125] In operation S1120, the first image features are processed using a classification function to obtain intermediate recognition results for N characters to be recognized. The classification function is used to classify the first image features.

[0126] In operation S1130, the first image features and the first sequence of text are input into the decoder of the character recognition model to obtain the recognition results of N characters to be recognized output by the decoder. The first sequence of text is obtained by right-shifting each character recognition result in the intermediate recognition result by M character positions, where M is an integer greater than or equal to 1.

[0127] Reference Figure 4 The intermediate recognition results can correspond to the first prediction tensor. Unlike the right-shifted text obtained from the labeled text during the training phase, the intermediate recognition results are used as the right-shifted text and input into the decoder along with the first image features. The decoder is configured to reorder the intermediate recognition results based on the first image features and output the final recognition result, thereby improving the recognition accuracy.

[0128] In some embodiments, a cross-attention layer is used to process the first image features and the first sequence of text to obtain first attention features, wherein the cross-attention layer is configured to process the data based on a cross-attention mechanism. The first attention features are then processed by a classification layer to obtain recognition results for N characters to be recognized.

[0129] Figure 12 A flowchart illustrating the process of obtaining the recognition results of N characters to be recognized according to another embodiment of this disclosure is shown. Figure 12 As shown, this embodiment is one example of operation S910, obtaining N characters to be recognized includes:

[0130] In operation S1210, the encoder of the character recognition model processes the image to be recognized and obtains the first image features output by the encoder.

[0131] In operation S1220, the first image features are input into the decoder of the character recognition model to obtain the recognition results of N characters to be recognized output by the decoder.

[0132] and Figure 11 The difference between the embodiments shown is that, Figure 12 The illustrated embodiment does not obtain intermediate recognition results; it only inputs the first image features into the decoder, resulting in fast recognition speed and sufficient accuracy. [CLS] can be used as... Figure 4 The text shifted right is input into the decoder along with the first image features. The [CLS] special encoding marker indicates the start of the text sequence. It should be understood that [CLS] is merely an example and the marker can be flexibly set.

[0133] Reference Figure 11 and Figure 12Two different methods for obtaining recognition results can be considered based on the first weight "α" and the second weight "1-α" in Equation 1 above. Figure 11 or Figure 12 If α is large and the desired model prediction accuracy is achieved, a CTC beam search can be performed first using the CTC function, and then the intermediate recognition results can be input into the decoder for decoding using the AttentionRescore method. Figure 12 This can be achieved by using a method where α is small and the speed of model prediction is prioritized, such as simply feeding the input to the decoder and performing decoding using an Attention bundle search. Figure 11 In this way.

[0134] Figure 13 A flowchart illustrating an end-to-end special character recognition method based on YOLO and TrOCR according to embodiments of the present disclosure is shown schematically. (See also: Joint Reference) Figures 1-12 and any of the above embodiments, Figure 13 The end-to-end special character recognition methods shown include:

[0135] In operation S1310, a YOLO-V7 object detection model is trained to detect character structures, and then the image containing only the entire character structure is cropped out. A character structure is a target region containing the character to be identified, which can be represented as a target region bounding box.

[0136] In operation S1320, a character recognition model is built based on TrOCR. Its Encoder part can be any Vision Transformer model or image feature extraction model, such as Swin Transformer, CSwin Transformer, etc., and its Decoder part can be the Transformer Decoder algorithm.

[0137] In operation S1330, a weighted hybrid loss function based on the CTC loss function and the classification loss function is constructed as the third loss function.

[0138] In operation S1340, the constructed TroOCR character recognition model is trained. The model is input with a character structure image, a character sequence ID shifted one position to the right, and label data (character sequence ID). The weighted mixture loss function constructed in S1330 is used, and the model is trained using optimizers such as Adam or AdamW. During model training, a series of data augmentation processes can be performed on the image data, such as Gaussian blur, random masking, image translation, and adjustments to image color, saturation, and brightness.

[0139] During the training phase, to accelerate model convergence, pre-trained model parameters for image classification can be loaded into the encoder, and pre-trained model parameters for natural language processing, such as ERNIE 3.0 and BERT, can be loaded into the decoder. Additionally, to achieve better model generalization and reduce manual annotation costs, a large number of simulated labeled samples can be synthesized using computer programs, and then fine-tuned using a small amount of manually labeled real data.

[0140] When operating the S1350, the trained TroOCR model is given an image with an unknown character sequence as input. The Attention bundle search method is used to decode the image and predict the character sequence, which is then verified.

[0141] In operation S1360, the object detection model and the character recognition model are concatenated to form an end-to-end recognition model. In end-to-end application, the entire document image is input to the end-to-end recognition model. The object detection model obtains slices of the character structure to get the image to be recognized. Then, the slice character recognition model decodes the slices to predict the character sequence on the slices.

[0142] According to embodiments of this disclosure, an end-to-end character recognition scheme is proposed, which can effectively recognize irregularly arranged characters and symbols, and features high accuracy and robustness. It can be effectively used for header recognition, formula recognition, and multi-line text recognition of various seals such as round seals, oval seals, and square seals.

[0143] It is understood that the object detection model used in this embodiment can be any object model. The Encoder used in the TroOCR algorithm of this invention can be any image feature extraction model, and the Decoder part can be any natural language model based on Transformer Encoder / Decoder. The classification loss function in the hybrid loss function used in this invention can also be any classification loss function.

[0144] Based on the above-mentioned character recognition model training method and character recognition method, this disclosure also provides a character recognition model training device and a character recognition device. The following will be combined with... Figure 14 and Figure 15 The device is described in detail.

[0145] Figure 14 A schematic block diagram of a training apparatus for a character recognition model according to an embodiment of the present disclosure is shown.

[0146] like Figure 14As shown, the character recognition model training device 1400 of this embodiment includes a first input module 1410, a first prediction module 1420, a second input module 1430, a loss calculation module 1440, a comprehensive loss module 1450, and a model update module 1460.

[0147] The first input module 1410 can perform operation S210, which is used to input the training image into the encoder of the character recognition model to obtain the second image features output by the encoder. The training image includes N characters to be recognized in an irregular arrangement, and the irregular arrangement includes not being arranged in a straight line. N is an integer greater than or equal to 2.

[0148] The first prediction module 1420 can perform operation S220 to obtain a first prediction tensor based on the second image features. The first prediction tensor includes character prediction information for N characters to be recognized.

[0149] The second input module 1430 can perform operation S230 to input the second image features into the decoder of the character recognition model and obtain the second prediction tensor output by the decoder. The second prediction tensor includes character prediction information for N characters to be recognized.

[0150] In some embodiments, the second input module 1430 may also perform operations S710 to S730, which will not be described in detail here.

[0151] The loss calculation module 1440 can perform operation S240 to obtain a first loss function value between the first prediction tensor and the labeled text, and a second loss function value between the second prediction tensor and the labeled text, wherein the labeled text includes a label for each character to be identified.

[0152] The comprehensive loss module 1450 can perform operation S250 to obtain the comprehensive loss function value based on the first loss function value and the second loss function value.

[0153] In some embodiments, the second input module 1430 can also perform operations S810 to S830, which will not be described in detail here.

[0154] The model update module 1460 can perform operation S260 to update the encoder and decoder based on the comprehensive loss function value to obtain the trained character recognition model.

[0155] In some embodiments, the training device 1400 may further include a first target detection module, which is used to perform operations S510 to S530, which will not be described in detail here.

[0156] In particular, the training device 1400 includes components for performing the above-mentioned tasks. Figures 2-8 The modules of each step in any of the described embodiments.

[0157] Figure 15 A schematic block diagram of a character recognition device according to an embodiment of the present disclosure is shown.

[0158] like Figure 15 As shown, the character recognition device 1500 of this embodiment includes a third input module 1510 and a recognition result module 1520. In some embodiments, the character recognition device 1500 may further include an image acquisition module for acquiring an image to be recognized. Figure 15 As shown in the image.

[0159] The third input module 1510 can perform operation S910 to input the image to be recognized into the character recognition model, wherein the character recognition model is obtained according to the training method described in any of the above embodiments, the image to be recognized includes N characters to be recognized in an irregular arrangement, the irregular arrangement includes not arranged in a straight line, and N is an integer greater than or equal to 2.

[0160] The recognition result module 1520 can perform operation S930 to obtain the recognition results of N characters to be recognized output by the character recognition model.

[0161] The character recognition model is configured to be pre-trained through the following operations: obtaining a first loss function value based on the encoder's encoding result, obtaining a second loss function value based on the decoder's decoding result, and updating the encoder and decoder based on the first and second loss function values. Specifically, see, for example... Figures 2-8 The training method described in any of the illustrated embodiments.

[0162] In some embodiments, the identification result module 1520 may also perform operations S1110 to S1130, or operations S1210 to S1220, which will not be described in detail here.

[0163] In some embodiments, the character recognition device 1500 may further include a second target detection module, which is used to perform operations S1010 to S1030, and will not be described in detail here.

[0164] In particular, the character recognition device 1500 includes components for performing the above-mentioned functions. Figures 9-12 The modules of each step in any of the described embodiments.

[0165] It should be noted that the implementation methods, technical problems solved, functions achieved, and technical effects of each module / unit / subunit in the device embodiments are the same as or similar to the implementation methods, technical problems solved, functions achieved, and technical effects of each corresponding step in the method embodiments, and will not be repeated here.

[0166] According to embodiments of this disclosure, any plurality of modules in the training device 1400 or the character recognition device 1500 may be combined into one module, or any one of these modules may be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules may be combined with at least some of the functions of other modules and implemented in one module.

[0167] According to embodiments of this disclosure, at least one of the training device 1400 or the character recognition device 1500 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three methods. Alternatively, at least one of the training device 1400 or the character recognition device 1500 can be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.

[0168] Figure 16 A block diagram of an electronic device suitable for implementing a training method for a character recognition model or a character recognition method according to embodiments of the present disclosure is shown schematically.

[0169] like Figure 16 As shown, an electronic device 1600 according to an embodiment of the present disclosure includes a processor 1601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1602 or a program loaded from a storage portion 1608 into a random access memory (RAM) 1603. The processor 1601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1601 may also include onboard memory for caching purposes. The processor 1601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0170] RAM 1603 stores various programs and data required for the operation of electronic device 1600. Processor 1601, ROM 1602, and RAM 1603 are interconnected via bus 1604. Processor 1601 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 1602 and / or RAM 1603. It should be noted that programs may also be stored in one or more memories other than ROM 1602 and RAM 1603. Processor 1601 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in one or more memories.

[0171] According to embodiments of this disclosure, the electronic device 1600 may further include an input / output (I / O) interface 1605, which is also connected to a bus 1604. The electronic device 1600 may also include one or more of the following components connected to the I / O interface 1605: an input section 1606 including a keyboard, mouse, etc.; an output section 1607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1608 including a hard disk, etc.; and a communication section 1609 including a network interface card such as a LAN card, modem, etc. The communication section 1609 performs communication processing via a network such as the Internet. A drive 1610 is also connected to the I / O interface 1605 as needed. A removable medium 1611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 1610 as needed so that computer programs read from it can be installed into the storage section 1608 as needed.

[0172] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0173] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 1602 and / or RAM 1603 and / or one or more memories other than ROM 1602 and RAM 1603 described above.

[0174] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement the methods provided in the embodiments of this disclosure.

[0175] When the computer program is executed by the processor 1601, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0176] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 1609, and / or installed from a removable medium 1611. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0177] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 1609, and / or installed from the removable medium 1611. When the computer program is executed by the processor 1601, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0178] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0179] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0180] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0181] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A character recognition method, comprising: Obtain an image to be recognized, wherein the image to be recognized includes N characters to be recognized in an irregular arrangement, the irregular arrangement including those not arranged in a straight line, and N is an integer greater than or equal to 2; Input the image to be recognized into the character recognition model; Obtain the recognition results of the N characters to be recognized output by the character recognition model; The character recognition model includes an encoder and a decoder, and is configured to be pre-trained through the following operations: A first loss function value is obtained based on the encoding result of the encoder, a second loss function value is obtained based on the decoding result of the decoder, and the encoder and the decoder are updated based on the first loss function value and the second loss function value. The recognition results of the N characters to be recognized output by the character recognition model include: The encoder is used to process the image to be recognized to obtain the first image feature output by the encoder, wherein the encoder divides the image to be recognized into blocks, flattens them into a sequence, and inputs the encoded part to obtain the first image feature; The first image features are processed using a classification function to obtain intermediate recognition results for the N characters to be recognized. The classification function is used to classify the first image features. The first image features and the first sequence text are input into the decoder to obtain the recognition results of the N characters to be recognized output by the decoder. The first sequence text is obtained by right-shifting the recognition result of each character in the intermediate recognition result by M character positions, where M is an integer greater than or equal to 1.

2. The method according to claim 1, wherein, The acquisition of the image to be identified includes: Identify the target image; A first target region is determined from the target image using a target detection model, the first target region including the N characters to be identified; The first target region is segmented from the target image to obtain the image to be identified.

3. The method according to claim 1, wherein, The decoder includes a cross-attention layer and a classification layer. The step of inputting the first image features and the first sequence of text into the decoder to obtain the recognition results of the N characters to be recognized output by the decoder includes: The first image features and the first sequence text are processed using the cross-attention layer to obtain the first attention features, wherein the cross-attention layer is configured to process data based on the cross-attention mechanism; The classification layer is used to process the first attention feature to obtain the recognition results of the N characters to be recognized. The classification layer is used to classify the first attention feature.

4. The method according to claim 1, wherein, The encoding result includes second image features, and obtaining the first loss function value based on the encoding result of the encoder includes: The training image is input into the encoder to obtain the second image features output by the encoder, wherein the training image includes N characters to be recognized in an irregular arrangement; A first prediction tensor is obtained based on the second image features. The first prediction tensor includes character prediction information for the N characters to be identified. Obtain the first loss function value between the first prediction tensor and the labeled text, wherein the labeled text includes a label for each character to be identified.

5. The method according to claim 4, wherein, The decoding result includes a second prediction tensor, and the second loss function value obtained based on the decoding result of the decoder includes: The second image feature is input into the decoder to obtain the second prediction tensor output by the decoder. The second prediction tensor includes character prediction information for the N characters to be identified. Obtain the second loss function value between the second predicted tensor and the labeled text.

6. The method according to claim 5, wherein, The step of updating the encoder and the decoder based on the first loss function value and the second loss function value includes: Based on the first loss function value and its first weight, the first weighted value is obtained; The second weighted value is obtained based on the second loss function value and its second weight; The comprehensive loss function value is obtained based on the first weighted value and the second weighted value; The encoder and decoder are updated based on the comprehensive loss function value to obtain the trained character recognition model.

7. The method according to claim 5, wherein, The decoder includes a cross-attention layer and a classification layer. The step of inputting the second image features into the decoder to obtain the second prediction tensor output by the decoder includes: The second image features and the second sequence text are input into the decoder. The second sequence text is obtained by right-shifting the label of each character to be identified in the label text by M character positions, where M is an integer greater than or equal to 1. The second image features and the second sequence text are processed using the cross-attention layer to obtain the second attention features, wherein the cross-attention layer is configured to process data based on the cross-attention mechanism; The second attention feature is processed using the classification layer to obtain the second prediction tensor.

8. The method according to claim 4, wherein, The step of obtaining the first prediction tensor based on the second image features includes: The second image features are then subjected to dimensionality upscaling to obtain the third image features; The third image feature is input into the classification function to obtain the first prediction tensor.

9. A character recognition device, comprising: An image acquisition module is used to acquire an image to be recognized, wherein the image to be recognized includes N characters to be recognized in an irregular arrangement, the irregular arrangement including those not arranged in a straight line, and N is an integer greater than or equal to 2; The third input module is used to input the image to be recognized into the character recognition model; The recognition result module is used to obtain the recognition results of the N characters to be recognized output by the character recognition model; The character recognition model includes an encoder and a decoder, and is configured to be pre-trained through the following operations: A first loss function value is obtained based on the encoding result of the encoder, a second loss function value is obtained based on the decoding result of the decoder, and the encoder and the decoder are updated based on the first loss function value and the second loss function value. The recognition results of the N characters to be recognized output by the character recognition model include: The encoder is used to process the image to be recognized to obtain the first image feature output by the encoder, wherein the encoder divides the image to be recognized into blocks, flattens them into a sequence, and inputs the encoded part to obtain the first image feature; The first image features are processed using a classification function to obtain intermediate recognition results for the N characters to be recognized. The classification function is used to classify the first image features. The first image features and the first sequence text are input into the decoder to obtain the recognition results of the N characters to be recognized output by the decoder. The first sequence text is obtained by right-shifting the recognition result of each character in the intermediate recognition result by M character positions, where M is an integer greater than or equal to 1.

10. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors perform the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 8.

12. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 8.

Images