Text recognition method, device, storage medium and program product

Abstract

This application discloses a recognition method, device, storage medium, and program product, relating to the field of artificial intelligence technology. The method includes: using a large model to recognize the text content and hierarchical structure of an image to be recognized, obtaining a word-by-word sequence. This word sequence contains the text content of the image to be recognized, and hierarchical labeling information for marking text units at different levels within the text content. The method further involves obtaining deep features of each sub-word sequence of the target level to be located. The deep features of each sub-word sequence include: hidden information extracted during the generation of the sub-word sequence by the large model; each sub-word sequence includes a set of labeling information for the target level and words within its labeled range; and for each sub-word sequence of the target level, predicting the target location coordinate sequence of the text units within that sub-word sequence in the image to be recognized based on the deep features of that sub-word sequence. This application improves text recognition efficiency.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a text recognition method, device, storage medium, and program product. Background Technology

[0002] In practical applications of text recognition tasks in the field of education, in order to effectively support downstream tasks (such as answer area recognition and accurate location of answer errors in homework grading scenarios, and content layout restoration of structured test paper documents), text recognition tasks must not only recognize the text content of each line of text, but also recognize the spatial location information of the text lines in the image.

[0003] Current text recognition tasks typically utilize large text recognition models (i.e., large OCR models) to perform text recognition on images. These large OCR models treat the coordinate information of text lines as special text tokens and output them together with the text content of the text lines. In other words, the output of a large OCR model is a mixed sequence of "text + coordinates". This text recognition scheme has low recognition efficiency.

[0004] Therefore, improving text recognition efficiency has become an urgent technical problem to be solved. Summary of the Invention

[0005] In view of the above problems, this application provides a text recognition method, device, storage medium, and program product to improve text recognition efficiency. The specific solution is as follows:

[0006] The first aspect of this application provides a text recognition method, including:

[0007] By using a large model to identify the text content and hierarchical structure of the image to be identified, a word-by-word sequence is obtained; the word sequence contains the text content of the image to be identified, as well as hierarchical labeling information for marking different levels of text units in the text content;

[0008] The deep features of each sub-word sequence of the target level to be located are obtained. The deep features of each sub-word sequence include: the hidden layer information extracted during the generation of the sub-word sequence by the large model; each sub-word sequence includes a set of hierarchical labeling information of the target level and the words within its labeled range.

[0009] For each sub-word sequence corresponding to the target level, the target location coordinate sequence of the text unit in the sub-word sequence is predicted in the image to be identified based on the deep features of the sub-word sequence.

[0010] In one possible implementation, text content and hierarchical structure recognition of the image to be recognized includes:

[0011] The image to be identified is encoded to obtain image features; autoregressive decoding is performed based on the image features to obtain the word sequence;

[0012] Deep feature prediction of target location coordinate sequence based on any sub-word sequence includes:

[0013] The target location coordinate sequence is obtained by performing non-autoregressive decoding based on the target features.

[0014] The target features include the image features and the deep features of any sub-word sequence;

[0015] Alternatively, the target feature may be a feature selected from a first feature and a second feature; the first feature is a deep feature of any sub-word sequence, and the second feature includes the image feature and the deep feature of any sub-word sequence.

[0016] In one possible implementation, non-autoregressive decoding is performed based on the image features and the deep features of any sub-word sequence, including:

[0017] The deep features of any sub-word sequence are fused to obtain the aggregated feature vector of any sub-word sequence;

[0018] Non-autoregressive decoding is performed on the aggregated feature vector of any sub-word sequence to obtain the initial positioning coordinate sequence corresponding to any sub-word sequence;

[0019] The image features, the aggregated feature vector, and the initial localization coordinate sequence are subjected to non-autoregressive decoding based on an attention mechanism to obtain the target localization coordinate sequence of the text unit in any sub-word sequence in the image to be identified; the accuracy of the target localization coordinate sequence is higher than that of the initial localization coordinate sequence.

[0020] In one possible implementation, non-autoregressive decoding based on the target features includes:

[0021] The deep features of any sub-word sequence are fused to obtain the aggregated feature vector of any sub-word sequence;

[0022] Based on the positioning accuracy requirements, the aggregated feature vector of any sub-word sequence is non-autoregressive decoded to obtain the target positioning coordinate sequence;

[0023] Alternatively, non-autoregressive decoding is performed on the aggregated feature vector of any sub-word sequence to obtain the initial positioning coordinate sequence corresponding to any sub-word sequence; non-autoregressive decoding based on an attention mechanism is performed on the image features, the aggregated feature vector, and the initial coordinate sequence to obtain the target positioning coordinate sequence; the accuracy of the target coordinate sequence is higher than that of the initial positioning coordinate sequence.

[0024] In one possible implementation, the text recognition method is implemented through a text recognition model, including:

[0025] The image encoding module of the text recognition model extracts features from the image to be recognized to obtain image features;

[0026] The word sequence is obtained by performing autoregressive decoding based on the image features through the autoregressive decoding module of the text recognition model; the autoregressive decoding module belongs to the large model.

[0027] The deep features of any sub-word sequence are fused by the fusion module of the text recognition model to obtain the aggregated feature vector of any sub-word sequence;

[0028] The text recognition model uses a non-autoregressive decoding module to perform non-autoregressive decoding based on target features to obtain the target location coordinate sequence; the non-autoregressive decoding module includes a lightweight non-autoregressive decoding module and an attention-based non-autoregressive decoding module.

[0029] In one possible implementation, the text recognition model is trained in the following manner:

[0030] The autoregressive decoding module was pre-trained in a self-supervised manner by adding a plain text corpus with hierarchical tagging information.

[0031] A pre-trained text detection and segmentation model is obtained by pre-training a first-class image sample labeled with text unit category, location information, and mask. The text detection and segmentation model includes: the image encoding module and the non-autoregressive decoding module based on the attention mechanism; each text unit belongs to a level.

[0032] The lightweight non-autoregressive decoding module, the fusion module, the pre-trained image encoding module, the autoregressive decoding module, and the attention-based non-autoregressive decoding module are jointly trained using second-class image samples of text based on labeled text unit location information, masks, and hierarchical labeling information to obtain the trained text recognition model. The accuracy of the location coordinate sequence decoded by the attention-based non-autoregressive decoding module is higher than that of the location coordinate sequence decoded by the lightweight non-autoregressive decoding module.

[0033] In one possible implementation, the text content includes text units from at least some of the following hierarchies:

[0034] Title level, paragraph level, illustration level, table level, text line level, character level;

[0035] The text units at the title level include text units at the paragraph level, text units at the illustration level, and text units at the table level; text units at the paragraph level contain text units at the line level; text units at the illustration level contain text units at the line level; text units at the table level contain text units at the line level; and text units at the text level contain text units at the character level.

[0036] A second aspect of this application provides a computer program product including computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the text recognition method of the first aspect or any implementation thereof.

[0037] A third aspect of this application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0038] The memory is used to store computer programs;

[0039] The processor is used to execute the computer program so that the electronic device can implement the text recognition method of the first aspect or any implementation thereof.

[0040] A fourth aspect of this application provides a computer storage medium carrying one or more computer programs that, when executed by an electronic device, enable the electronic device to implement the text recognition method described in the first aspect or any implementation thereof.

[0041] Using the above technical solutions, the text recognition method, device, storage medium, and program product provided in this application recognize the text content and hierarchical structure of the image to be recognized through a large model, obtaining a word-by-word generated word sequence. This word sequence contains the text content in the image to be recognized, as well as hierarchical labeling information for marking text units at different levels in the text content. It also obtains deep features of each sub-word sequence of the target level to be located. The deep features of each sub-word sequence include: hidden layer information extracted during the generation of the sub-word sequence by the large model; each sub-word sequence includes a set of hierarchical labeling information of the target level and words within its labeled range; and for each sub-word sequence of the target level, it predicts the target location coordinate sequence of the text units within that sub-word sequence in the image to be recognized based on the deep features of that sub-word sequence. In this application, the large model no longer identifies the target localization coordinate sequence of text content in the image to be identified, but only identifies the text content and the hierarchical structure of the text content in the image, corresponding to the target level to be located, and obtains the deep features of each sub-word sequence of the target level generated by the large model. Based on the deep features of each sub-word sequence, the target localization coordinate sequence of the text unit in the sub-word sequence is predicted in the image to be identified, so that the identification of text content and the prediction of target localization coordinate sequence can be carried out simultaneously, thereby improving the efficiency of text recognition. Attached Figure Description

[0042] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0043] Figure 1 A flowchart illustrating an implementation of the text recognition method provided in this application;

[0044] Figure 2 A flowchart illustrating an implementation of non-autoregressive decoding based on deep features of image features and any sub-word sequence provided in this application;

[0045] Figure 3 A schematic diagram of the structure of the text recognition model provided in this application;

[0046] Figure 4 A schematic diagram of the structure of the Mask DINO network provided in this application;

[0047] Figure 5 A schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0048] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0049] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0050] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0051] To improve the efficiency of text recognition from images, this application proposes a solution.

[0052] like Figure 1 The diagram shown is a flowchart of one implementation of the text recognition method provided in this application, which may include:

[0053] Step S101: The large model is used to identify the text content and hierarchical structure of the image to be recognized, resulting in a word-by-word sequence. The word sequence contains the text content of the image to be recognized, as well as hierarchical labeling information used to mark different levels of text units in the text content.

[0054] Large models can be large language models (LLMs) or multimodal large models (MLLMs).

[0055] The image to be identified can be from any of the following fields: education, aerospace, medical, legal, scientific research, etc.

[0056] If the large model is a large language model, the image to be recognized can be encoded by an image encoder to obtain image features; the large model can then use these image features to identify the text content and hierarchical structure of the text content in the image to be recognized.

[0057] If the large model is a multimodal large model, the image to be recognized can be directly input into the multimodal large model, and the image encoder of the multimodal large model encodes the image to be recognized to obtain image features; based on the image features, the text content and hierarchical structure of the text content in the image to be recognized can be obtained.

[0058] Depending on the application area, the number of levels can vary. Taking education as an example, multiple levels can include, but are not limited to: question level, paragraph level, illustration level, table level, text line level, and character level. Specifically, a question-level text unit can include text units from the paragraph, illustration, and table levels; a paragraph-level text unit contains text units from the text line level; an illustration-level text unit contains text units from the text line level; a table-level text unit contains text units from the text line level; and a text line level text unit contains text units from the character level. Each character-level text unit is a single character.

[0059] As an example, hierarchical labeling information in the field of education may include, but is not limited to:

[0060] <topic> and< / topic> This set of hierarchical markers is used to mark the beginning and end of a problem (or sub-problem), and supports nesting (e.g., ...). <topic>Question 1 <topic> Question 1.1< / topic> < / topic> This indicates that question 1 contains sub-question 1.1.

[0061] <paragraph> and< / paragraph> This set of hierarchical markers is used to mark the boundaries of a paragraph (such as paragraphs in an essay).

[0062] <line> and< / line> This set of hierarchical markers is used to mark the boundaries of a line of text;

[0063] <graph> and< / graph> This set of hierarchical markers is used to mark the boundaries of the descriptive text in the illustration area;

[0064] and This set of hierarchical markers is used to mark the boundaries of text in a table area.

[0065] The boundaries of a single character are well-defined and do not require hierarchical marking information. A single character is the most basic unit of text.

[0066] For example, after a math homework image is predicted by a large language model, the output word sequence might be: <topic> <line> Question 1: Calculate the following expressions< / line> <line> （1）2+3×5< / line> <line> （2）10÷2-3< / line> < / topic> <topic> Question 2: ...< / topic> 。

[0067] The first one <topic> With the first< / topic> Define the whole of "Problem 1" (i.e., "Problem 1: Calculate the following expressions (1) 2 + 3 × 5 (2) 10 ÷ 2 - 3"), the second<topic> With the second< / topic> Define "Question 2" as a whole; the first <line> and the first< / line > Define the line for the question "Question 1: Calculate the following expressions", the second line and the second< / line > Define the line of the expression “(1)2+3×5”, the third line and the third line< / line > Define the line of the expression “(2) 10÷2-3”.

[0068] For other domains, if there is no title level, multiple levels may include, but are not limited to: paragraph level, illustration level, table level, text line level, and character level; wherein, the text unit of the paragraph level contains the text unit of the text line level; the text unit of the illustration level contains the text unit of the text line level; the text unit of the table level contains the text unit of the text line level; and the text unit of the text line level contains the text unit of the character level.

[0069] Step S102: Obtain the deep features of each sub-word sequence of the target level to be located. The deep features of each sub-word sequence include: the hidden information extracted during the generation of the sub-word sequence by the large model; each sub-word sequence includes a set of hierarchical labeling information of the target level and the words within its labeled range.

[0070] In this application, because the text is divided into different hierarchical structures, the target level to be located can be specified according to the positioning accuracy requirements. For example, if it is necessary to locate the text line (e.g., in a homework grading scenario), the target level can be the text line level; if it is necessary to locate the question (e.g., in a structured textbook scenario), the target level can be the question level; if it is necessary to locate the character (e.g., in a word dictation scenario), the target level can be the character level.

[0071] Using the aforementioned lexical sequence " <topic> <line> Question 1: Calculate the following expressions< / line> <line> （1）2+3×5< / line> <line> （2）10÷2-3< / line> < / topic> <topic> Question 2: ...< / topic> For example, if the target level is the question level, then the set of level markers for the question level is: <topic> and< / topic> This example lexical sequence includes two question-level sub-lexical sequences, where the first question-level sub-lexical sequence is " <topic> <line> Question 1: Calculate the following expressions< / line> <line> （1）2+3×5< / line> <line> （2）10÷2-3< / line> < / topic> The second question level sub-word sequence is " <topic> Question 2: ...< / topic> If the target level is the text line level, then the set of hierarchical markers for the text line level is: <line> and< / line> Therefore, the sub-word sequence corresponding to "Question 1" includes three text line-level sub-word sequences, where the first text line-level sub-word sequence is " <line> Question 1: Calculate the following expressions< / line> The second text line level sub-word sequence is " <line> （1）2+3×5< / line> The third text line level sub-word sequence is " <line> （2）10÷2-3< / line> ".

[0072] It should be noted that if the target level is a character level, the level marker information of the target level is empty, that is, there is no level marker information.

[0073] The hidden information extracted during the generation of each sub-word sequence by the large model includes: the hidden information extracted during the generation of each word in the sub-word sequence by the large model.

[0074] The hidden information extracted for each token may include, but is not limited to, at least one of the following: the token embedding obtained from the previous decoding time of the token, the hidden state of the large model at the decoding time of the token, and the attention positions of the large model when performing attention calculation.

[0075] The large model performs attention calculations by using the hidden state as the query and each feature vector in the image features of the image to be identified as the key and value. Based on this, the attention position is the location of each feature vector within the image features.

[0076] Step S103: For each sub-word sequence corresponding to the target level, predict the target location coordinate sequence of the text unit in the sub-word sequence in the image to be identified based on the deep features of the sub-word sequence.

[0077] The target localization coordinate sequence of a text unit in any sub-word sequence in the image to be recognized can be the coordinates of the four vertices of the bounding box of the text unit in any sub-word sequence. Based on this, there are 8 coordinate values ​​in the target localization coordinate sequence, which can be denoted as: (x1,y1,x2,y2,x3,y3,x4,y4).

[0078] Alternatively, the target localization coordinate sequence of a text unit in any sub-word sequence in the image to be recognized can be the coordinates of the two diagonal vertices of the bounding box of the text unit in any sub-word sequence. Based on this, there are 4 coordinate values ​​in the target localization coordinate sequence, which can be denoted as (x1, y1, x2, y2).

[0079] The coordinate values ​​in the target positioning coordinate sequence can be generated simultaneously or sequentially, one coordinate value at a time.

[0080] Compared with the existing technology (which uses coordinate values ​​as tokens, generates the target location coordinate sequence of the i-th text line word by word after generating the i-th text line word by word, then generates the (i+1)-th text line word by word, and so on, until the text recognition ends), this application, based on the generation of the i-th sub-token sequence at the target level, can predict the target location coordinate sequence of the text unit in the i-th sub-token sequence in the image to be recognized based on the deep features of the i-th sub-token sequence, after generating the i-th sub-token sequence at the target level and at the same time as generating the (i+1)-th sub-token sequence at the target level. This achieves parallel generation of text units and coordinates, resulting in higher recognition efficiency.

[0081] In addition, although this application requires generation of hierarchical tag information per word, the amount of data in the hierarchical tag information is much smaller than the amount of data in the target positioning coordinate sequence relative to the bounding box of the text line. Moreover, the recognition of text content and the prediction of the target positioning coordinate sequence can be performed simultaneously. Therefore, it will not reduce the efficiency of text recognition, but rather improve the efficiency of text recognition.

[0082] Using the sub-word sequence as " <topic> <line> Question 1: Calculate the following expressions< / line> <line> （1）2+3×5< / line> <line> （2）10÷2-3< / line> < / topic> For example, the target location coordinate sequence predicted by the deep features of the sub-word sequence is "Question 1: Calculate the coordinates of the vertices (four vertices or two diagonal vertices) of the bounding box of the following text unit (1) 2+3×5 (2) 10÷2-3" in the image to be identified.

[0083] Using the sub-word sequence as " <line> （1）2+3×5< / line> For example, the target location coordinate sequence predicted based on the deep features of the sub-word sequence is the coordinates of the vertices (four vertices or two diagonal vertices) of the bounding box of the text unit “(1)2+3×5” in the image to be identified.

[0084] The text recognition method provided in this application embodiment no longer identifies the target location coordinate sequence of text content in the image to be recognized by the large model. Instead, it only identifies the text content and the hierarchical structure of the text content in the image, corresponding to the target level to be located. It obtains the deep features of each sub-word sequence of the target level extracted by the large model, and predicts the target location coordinate sequence of the text unit in the sub-word sequence in the image to be recognized based on the deep features of each sub-word sequence. This allows the recognition of text content and the prediction of target location coordinate sequence to be performed simultaneously, thereby improving the efficiency of text recognition.

[0085] In other words, this application achieves decoupled prediction of text content and coordinates through a three-stage process of "text prediction - unit splitting - coordinate regression", which improves inference efficiency while ensuring accuracy.

[0086] In an optional embodiment, one way to implement the above-described text content and hierarchical structure recognition of the image to be recognized can be:

[0087] The image to be recognized is encoded to obtain image features. Optionally, features can be extracted from the image to be recognized at multiple scales to obtain multi-scale features; these multi-scale features are then encoded to obtain enhanced multi-scale features as image features.

[0088] Autoregressive decoding is performed based on image features to generate word sequences word by word. In other words, each word in the word sequence is generated based on the words that have already been decoded.

[0089] Correspondingly, one implementation method for predicting target location coordinate sequences based on deep features of any sub-word sequence can be:

[0090] Non-autoregressive decoding is performed based on target features to obtain a target location coordinate sequence. In other words, each coordinate value in the target location coordinate sequence is generated independently, and the generation of each coordinate value does not depend on other coordinates. Therefore, all coordinate values ​​can be generated synchronously, thereby improving the efficiency of coordinate recognition, which in turn improves the overall efficiency of text recognition.

[0091] The target features include image features and deep features of any sub-word sequence.

[0092] Alternatively, the target feature can be selected from the first feature (the deep features of any sub-word sequence) and the second feature (including image features and the deep features of any sub-word sequence). In other words, the target feature is not fixed, but can be selected from the first and second features as the target feature based on requirements.

[0093] The accuracy of the target location coordinate sequence obtained by non-autoregressive decoding based on the second feature is higher than that of the target location coordinate sequence obtained by non-autoregressive decoding based on the first feature.

[0094] In this embodiment of the application, there are two ways to obtain the target positioning coordinate sequence corresponding to any sub-word sequence. One way is to directly perform non-autoregressive decoding based on image features and the deep features of any sub-word sequence without considering the accuracy requirements. The other way is to select a feature corresponding to the positioning accuracy requirements from the first feature and the second feature for non-autoregressive decoding according to the positioning accuracy requirements.

[0095] In an optional embodiment, the flowchart of one implementation of non-autoregressive decoding based on deep features of image features and any sub-word sequence is as follows: Figure 2 As shown, it may include:

[0096] Step S201: Fuse the deep features of any sub-word sequence to obtain the aggregated feature vector of that sub-word sequence.

[0097] Optionally, the deep features of any sub-word sequence can be fused using a fusion module (e.g., a separate Transformer layer) to obtain an aggregated feature vector, which contains information such as the semantics and hierarchical structure of the text units within any sub-word sequence.

[0098] When the deep features of any sub-word sequence include each attention position when the large model performs attention calculation, each attention position can be encoded first (e.g., sinusoidal position encoding) to obtain the embedding vector of each attention position, and then the embedding vector of each attention position can be input into the fusion module.

[0099] Step S202: Perform non-autoregressive decoding on the aggregated feature vector of any sub-word sequence to obtain the initial positioning coordinate sequence corresponding to any sub-word sequence.

[0100] A lightweight regression network can be used to perform non-autoregressive decoding on the aggregated feature vector of any sub-word sequence to obtain the initial positioning coordinate sequence corresponding to the sub-word sequence. This initial positioning coordinate sequence represents the low-precision position of the text unit in the sub-word sequence in the image to be recognized.

[0101] As an example, a lightweight regression network can consist of two fully connected layers and an activation function layer (such as the ReLU activation function), which can directly output the initial localization coordinate sequence, that is, the coordinate values ​​in the initial localization coordinate sequence are output synchronously.

[0102] Step S203: Perform non-autoregressive decoding based on an attention mechanism on the image features, aggregated feature vector, and initial localization coordinate sequence to obtain the target localization coordinate sequence of text units within any sub-word sequence in the image to be recognized. The accuracy of the target localization coordinate sequence is higher than that of the initial localization coordinate sequence.

[0103] The aggregation features of the sub-word sequence contain the content features of the text unit. Therefore, this application uses the content features of the text unit as a conditional input and directly predicts the floating-point coordinates of the text unit through a regression network to eliminate quantization error and thus achieve higher positioning accuracy.

[0104] Optionally, the initial positioning coordinate sequence can be encoded to obtain a coordinate feature vector. The coordinate feature vector and the aggregated feature vector can be fused (e.g., concatenated or added) to obtain a fused feature vector. The fused feature vector can be used as a query, and each feature vector in the image features can be used as a key and value for attention calculation. Based on the attention calculation result, non-autoregressive decoding can be performed to obtain the target positioning coordinate sequence.

[0105] In addition, since the fused feature vector corresponds one-to-one with the sub-word sequence, the decoding result can be directly aligned with the label, eliminating the need for matching using the Hungarian algorithm in traditional detection, further reducing computational overhead and accelerating convergence.

[0106] In an optional embodiment, one implementation of the above-described non-autoregressive decoding based on target features can be:

[0107] The deep features of any sub-word sequence are fused to obtain the aggregated feature vector of that sub-word sequence. For details on the implementation, please refer to the aforementioned embodiments; they will not be repeated here.

[0108] Based on the required positioning accuracy, select the method corresponding to the required positioning accuracy from the following two methods to obtain the target positioning coordinate sequence:

[0109] Method 1: Perform non-autoregressive decoding on the aggregated feature vector of any sub-word sequence to obtain the target location coordinate sequence.

[0110] Method 2: Perform non-autoregressive decoding on the aggregated feature vector of any sub-word sequence to obtain the initial positioning coordinate sequence corresponding to any sub-word sequence; perform non-autoregressive decoding on the image features, aggregated feature vector, and initial coordinate sequence based on an attention mechanism to obtain the target positioning coordinate sequence; the accuracy of the target coordinate sequence is higher than that of the initial positioning coordinate sequence.

[0111] Optionally, the positioning accuracy requirement can be a positioning accuracy range corresponding to the target level. Different positioning accuracy ranges may correspond to the same or different methods. The positioning accuracy range can include a first positioning accuracy range and a second positioning accuracy range. The maximum positioning accuracy in the first positioning accuracy range is less than the minimum positioning accuracy in the second positioning accuracy range. Based on this, the first positioning accuracy range corresponds to method one, and the second positioning accuracy range corresponds to method two. Therefore, users only need to specify the target level to automatically determine the positioning accuracy range, making the user operation relatively simple.

[0112] Optionally, the positioning accuracy requirement can be a range specified by the user in the positioning accuracy range list. This list can include a first positioning accuracy range and a second positioning accuracy range. The maximum positioning accuracy within the first range is less than the minimum positioning accuracy within the second range. Therefore, the first positioning accuracy range corresponds to method one, and the second positioning accuracy range corresponds to method two. To facilitate user selection, the positioning accuracy range list can provide two options: "High Accuracy" (corresponding to the second positioning accuracy range) and "General Accuracy" (corresponding to the first positioning accuracy range).

[0113] In an optional embodiment, the text recognition method can be implemented using a text recognition model. A schematic diagram of a text recognition model provided in this application embodiment is shown below. Figure 3 The term may include:

[0114] Image encoding module 301 (may belong to the large model or not), autoregressive decoding module 302 (may belong to the large model), fusion module 303 and non-autoregressive decoding module 304;

[0115] The image encoding module 301, also known as the image encoder, is used to extract features from the image to be recognized and obtain image features.

[0116] The autoregressive decoding module 302 (belonging to the large model) is used to perform autoregressive decoding based on image features to obtain a word sequence containing text content and hierarchical tagging information.

[0117] The fusion module 303 is used to fuse the deep features of any sub-word sequence to obtain the aggregated feature vector of the sub-word sequence.

[0118] The non-autoregressive decoding module 304 is used to perform non-autoregressive decoding based on target features to obtain the target location coordinate sequence; the non-autoregressive decoding module 304 includes a lightweight non-autoregressive decoding module 3041 and an attention-based non-autoregressive decoding module 3042.

[0119] The lightweight non-autoregressive decoding module 3041 is used to perform non-autoregressive decoding on the aggregated feature vector of any sub-word sequence to obtain the initial positioning coordinate sequence corresponding to any sub-word sequence.

[0120] The attention-based non-autoregressive decoding module 3042 is used to perform attention-based non-autoregressive decoding on image features, aggregated feature vectors and initial localization coordinate sequences to obtain the target localization coordinate sequence of text units in any sub-word sequence in the image to be recognized.

[0121] In this application, the autoregressive decoding module 302 (belonging to a large model) and the non-autoregressive decoding module 304 share the image encoding module 301. This means that content recognition and coordinate prediction share contextual features, maintaining semantic consistency and achieving a "what you see is what you get" structured output. This provides efficient and more accurate spatial perception capabilities for multimodal tasks such as educational OCR, document intelligence, and visual question answering.

[0122] In an optional embodiment, the text recognition model can be trained in the following manner:

[0123] First, the autoregressive decoding module 302 (belonging to the large model), the image encoding module 301, and the attention-based non-autoregressive decoding module 3042 are pre-trained. Then, the lightweight non-autoregressive decoding module 3041, the fusion module 303, the pre-trained image encoding module 301, the pre-trained autoregressive decoding module 302, and the pre-trained attention-based non-autoregressive decoding module 3042 are jointly trained to obtain the trained text recognition model.

[0124] The pre-training process for the autoregressive decoding module 302 (which belongs to a large model) may include:

[0125] The autoregressive decoding module 302 is pre-trained by adding hierarchical labeling information to plain text corpora, resulting in a pre-trained autoregressive decoding module 302. This pre-trained autoregressive decoding module 302 has the ability to recognize text and generate hierarchical structures (i.e., correctly output hierarchical labeling information, as described above). <topic> 、< / topic> , <line> 、< / line> (Hierarchical labeling information), but does not involve coordinate information.

[0126] Plain text corpora may include, but are not limited to, the text content of textbooks, the text content of test papers, the text content of assignments, etc. This application adds different hierarchical tagging information to the aforementioned plain text corpora to mark different hierarchical structures within the plain text corpora.

[0127] When pre-training the autoregressive decoding module 302, the N+1th word in the plain text corpus with added hierarchical tagging information can be predicted based on the first N (N=1, 2, 3, ..., K-1; K is the total number of words in the plain text corpus with added hierarchical tagging information) words. The goal is to make the N+1th word predicted by the autoregressive decoding module 302 approach the N+1th word in the plain text corpus with added hierarchical tagging information, and the parameters of the autoregressive decoding module 302 are updated accordingly.

[0128] The pre-training process for the image encoding module 301 and the attention-based non-autoregressive decoding module 3042 may include:

[0129] A pre-trained text detection and segmentation model is obtained by pre-training the model using first-class image samples labeled with text unit categories, location information, and masks. This model includes an image encoding module 301 and an attention-based non-autoregressive decoding module 3042. Each text unit belongs to one level.

[0130] Text units can be categorized, including but not limited to, the following: printed text, handwritten text, illustrations, tables, answer areas, question numbers, large question boxes, and small question boxes. The location information of a text unit refers to the vertex coordinates of its bounding box in the image. The mask of a text unit refers to the mask of each word within that unit, used to segment pixel-level text unit regions from the image.

[0131] The text detection and segmentation model provided in this application embodiment can adopt a Mask DINO network structure, such as... Figure 4 The diagram shown is a structural schematic of a Mask DINO network provided in an embodiment of this application, wherein:

[0132] Figure 4 The CNN network and Encoder Layers × N constitute the image encoding module 301; Decoder Layers × M, the Hybrid Matching algorithm, and the mask generation branch (including Unflatten, 2×upsample, ...) and The non-autoregressive decoding module 3042, based on an attention mechanism, is constructed. The CNN network is used to extract multi-scale features from the image to be recognized, and the Encoder Layers×N are used to encode some of the multi-scale features to obtain enhanced multi-scale features. The Encoder Layers×N is a Transformer encoder.

[0133] The Unified Query Selection module is used to select feature vectors that meet certain conditions (Init Contents) from the feature vectors of the image features output by the image encoding module 301.

[0134] Decoder Layers×M and the Hybrid Matching algorithm are used to perform non-autoregressive decoding based on the image features (Key & Value) output by the image encoding module 301 and the conditional feature vectors (Init Contents) and their location information (Init Anchors) to obtain the location information (Boxes) and class (Classes) of each text unit in the first class of image samples of the query embedding. The location information (Init Anchors) of the conditional feature vectors refers to the position of the conditional feature vectors in the image features (i.e., feature map). DecoderLayers×M can be a Transformer decoder.

[0135] The enhanced multi-scale features are also used to upsample to obtain a 1 / 4 scale feature map, which is then fused (e.g., added) with the 1 / 4 scale features extracted from the input image to obtain a pixel embedding map. The pixel embedding map is then multiplied by the query embedding to obtain the mask of the text unit.

[0136] The training process for text detection and segmentation models can include:

[0137] The first type of image samples are input into the text detection and segmentation model. The CNN network extracts multi-scale features from the first type of image samples (the following example uses feature maps of 5 scales, which are 1 / 4, 1 / 8, 1 / 16, 1 / 32 and 1 / 64 of the input image, respectively).

[0138] Encoder Layers×N encodes features at certain scales (feature maps at scales other than the largest scale, i.e., 1 / 8, 1 / 16, 1 / 32, and 1 / 64 scales) and their positional embeddings (obtained by sinusoidal positional encoding of the positions of each feature vector in the feature map), resulting in enhanced features at four scales. The feature maps and positional embeddings at the four scales (1 / 8, 1 / 16, 1 / 32, and 1 / 64) are all flattened before being input into Encoder Layers×N.

[0139] The Unified Query Selection module performs preliminary predictions based on the enhanced four-scale features to obtain the text unit category and location information corresponding to each feature vector in the enhanced four-scale features. Based on the confidence score (or classification score) of the text unit category, it selects the top G feature vectors with the highest confidence scores in descending order as the feature vectors that meet the conditions. The feature vectors that meet the conditions and their corresponding predicted location information are then input into Decoder Layers×M.

[0140] Decoder Layers×M uses the feature vectors that meet the conditions and their location information fused (concatenated or added) as the Query. Attention is calculated using each feature vector in the multi-scale enhanced features as Key & Value. Combined with the Hybrid Matching algorithm, the query embedding, location information (Boxes), and categories (Classes) are obtained.

[0141] Additionally, the input to Decoder Layers×M can also include actual annotation information ( Figure 4 The noisy annotation information obtained by adding noise to the ground truth (GT, including text unit category, location information and mask) is ( Figure 4 GT+Noise (including noisy text unit categories, noisy location information, and noisy masks) is used in conjunction with the Unified DN component for training denoising tasks, enabling Decoder Layers×M to reconstruct location information, masks, and text unit categories in noisy conditions, thereby improving the robustness of the text recognition model.

[0142] The maximum scale (1 / 8) feature obtained after unflattening the enhanced multi-scale features is upsampled by 2× and then fused (added) with the original maximum scale (1 / 4) feature map to obtain the pixel embedding map.

[0143] The dot product of the query embedding and the pixel embedding map is used to obtain the mask of the text unit.

[0144] With the goal of ensuring that the text unit categories, location information, and masks predicted by the text detection and segmentation model are close to the labeled text unit categories, location information, and masks, the gradient descent algorithm is used to update the parameters of the text detection and segmentation model.

[0145] The above-mentioned text detection and segmentation model training process does not rely on text content information, enabling the text detection and segmentation model to have basic text region detection capabilities.

[0146] By independently pre-training the two modules, "autoregressive decoding module 302" and "image encoding module 301 and attention-based non-autoregressive decoding module 3042", it is ensured that the two modules respectively master the basic capabilities of "text content understanding" and "spatial coordinate localization".

[0147] After the text detection and segmentation model training is completed, the lightweight non-autoregressive decoding module 3041, the fusion module 303, the pre-trained image encoding module 301, the pre-trained autoregressive decoding module 302, and the pre-trained attention-based non-autoregressive decoding module 3042 are jointly trained using second-class image samples of text with labeled text unit location information, masks, and hierarchical labeling information to obtain a trained text recognition model. The accuracy of the location coordinate sequence decoded by the attention-based non-autoregressive decoding module 3042 is higher than that of the location coordinate sequence decoded by the lightweight non-autoregressive decoding module 3041. In other words, the attention-based non-autoregressive decoding module 3042 is used to correct the offset of the location coordinate sequence decoded by the lightweight non-autoregressive decoding module 3041. For example, in the homework grading scenario, the text line “(1)2+3×5” is obtained by the lightweight non-autoregressive decoding module 3041 with approximate coordinates (x1=100, y1=200, x2=300, y2=230). After the approximate coordinates are optimized by the attention-based non-autoregressive decoding module 3042, the precise coordinates (x1=102, y1=198, x2=298, y2=232) and mask (covering the pixel area of ​​the text line) are output to ensure that the highlighting is without deviation.

[0148] Optionally, the second type of image samples can be input into the text recognition model to obtain the word sequence output by the autoregressive decoding module 302, the aggregated feature vector of each text unit obtained by the fusion module 303, the initial localization coordinate sequence decoded by the lightweight non-autoregressive decoding module 3041, and the target localization coordinate sequence decoded by the attention-based non-autoregressive decoding module 3042.

[0149] The algorithm obtains the text loss of the word sequence output by the autoregressive decoding module 302 relative to the text with hierarchical labeling information of the second type of image samples, the coordinate loss of the initial localization coordinate sequence decoded by the lightweight non-autoregressive decoding module 3041 relative to the localization information of the second type of image samples, the bounding box loss of the bounding box determined by the target localization coordinate sequence decoded by the attention-based non-autoregressive decoding module 3042 relative to the bounding box determined by the localization information of the second type of image samples, the mask loss of the mask decoded by the attention-based non-autoregressive decoding module 3042 relative to the mask of the second type of image samples, and the contrastive learning loss of the aggregated feature vector of each text unit obtained by the fusion module 302 and the image features (vectors) of the region where each text unit is located in the second type of image samples.

[0150] With the goal of reducing the aforementioned text loss, coordinate loss, bounding box loss, mask loss, and contrastive learning loss, the parameters of the text recognition model are updated.

[0151] Optionally, the text loss can be cross-entropy loss (used to optimize the generation accuracy of text content and hierarchical label information), the coordinate loss can be L1 loss or L2 loss (used to optimize coordinate accuracy, and L1 / L2 loss naturally has continuity, improving training stability without label smoothing), the bounding box loss can be GIoU loss (used to optimize bounding box accuracy), and the mask loss can be DICE loss (used to optimize mask segmentation accuracy). Contrastive learning loss is used to reduce semantic and spatial bias, making the aggregated feature vector of each text unit approach the feature vector of the region image containing that text unit in the second-class image samples, and away from the feature vector of the region image not containing that text unit in the second-class image samples; and the region image containing that text unit in the second-class image samples approaches the aggregated feature vector of that text unit, and away from the aggregated feature vectors of other text units.

[0152] Through joint training, the hierarchical labeling information generated by the large model is made more aligned with spatial segmentation requirements, ensuring that the hierarchical labeling information accurately corresponds to the actual text units. For example, ensuring... <line> and< / line> The markers accurately correspond to the actual text lines.

[0153] In summary, this application transforms autoregressively generated coordinate tokens into a parallel regression task based on text unit features. By combining hierarchical text structure and detection model optimization, it solves the quantization error and efficiency problems of existing technologies. During their research, the inventors of this application discovered that in existing technologies, coordinates, as tokens, account for approximately 20% of all predicted tokens in the large model (including text content in images and coordinates of text units). In this application, text units and coordinates are predicted in parallel, eliminating 20% ​​of the autoregressive coordinate generation steps and reducing inference time by approximately 15%.

[0154] Furthermore, this application supports coordinate prediction for multi-granularity text units (single character, text line, paragraph, title), adapting to different task requirements.

[0155] Corresponding to the method embodiments, this application also provides an electronic device. (See reference...) Figure 5 The diagram illustrates a structural schematic of an electronic device suitable for implementing embodiments of this application. The electronic device in these embodiments can be a terminal device (e.g., an in-vehicle infotainment system, a large-screen device, a mobile phone, a tablet computer, a laptop computer, a desktop computer, etc.) or a server (which can be a single server, a server cluster, or a cloud server, etc.). Figure 5 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0156] like Figure 5 As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. When the electronic device is powered on, the RAM 503 also stores various programs and data required for the operation of the electronic device. The processing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0157] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 508 including, for example, memory cards, hard drives, etc.; and communication devices 509. Communication device 509 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have instead.

[0158] This application also provides a computer program product including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the text recognition methods provided in this application.

[0159] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the text recognition methods provided in this application.

[0160] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0161] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0162] In the above embodiments, the functionality can be implemented entirely or partially through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented entirely or partially as a computer program product. Those skilled in the art can use different methods to implement the described functions for each specific solution, but such implementation should not be considered beyond the scope of this application.

[0163] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

[0164] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0165] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A text recognition method, characterized in that, include: By using a large model to identify the text content and hierarchical structure of the image to be recognized, a word-by-word generation sequence of word elements is obtained. The word sequence contains the text content in the image to be identified, as well as hierarchical labeling information for marking different levels of text units in the text content; The deep features of each sub-word sequence of the target level to be located are obtained. The deep features of each sub-word sequence include: the hidden layer information extracted during the generation of the sub-word sequence by the large model; each sub-word sequence includes a set of hierarchical labeling information of the target level and the words within its labeled range. For each sub-word sequence corresponding to the target level, the target location coordinate sequence of the text unit in the sub-word sequence is predicted in the image to be identified based on the deep features of the sub-word sequence.

2. The method according to claim 1, characterized in that, Text content and hierarchical structure recognition of the image to be recognized includes: The image to be identified is encoded to obtain image features; autoregressive decoding is performed based on the image features to obtain the word sequence; Deep feature prediction of target location coordinate sequence based on any sub-word sequence includes: The target location coordinate sequence is obtained by performing non-autoregressive decoding based on the target features. The target features include the image features and the deep features of any sub-word sequence; Alternatively, the target feature may be a feature selected from a first feature and a second feature; the first feature is a deep feature of any sub-word sequence, and the second feature includes the image feature and the deep feature of any sub-word sequence.

3. The method according to claim 2, characterized in that, Non-autoregressive decoding based on the image features and the deep features of any sub-word sequence includes: The deep features of any sub-word sequence are fused to obtain the aggregated feature vector of any sub-word sequence; Non-autoregressive decoding is performed on the aggregated feature vector of any sub-word sequence to obtain the initial positioning coordinate sequence corresponding to any sub-word sequence; The image features, the aggregated feature vector, and the initial localization coordinate sequence are subjected to non-autoregressive decoding based on an attention mechanism to obtain the target localization coordinate sequence of the text unit in any sub-word sequence in the image to be identified; the accuracy of the target localization coordinate sequence is higher than that of the initial localization coordinate sequence.

4. The method according to claim 2, characterized in that, Non-autoregressive decoding based on the target features includes: The deep features of any sub-word sequence are fused to obtain the aggregated feature vector of any sub-word sequence; Based on the positioning accuracy requirements, the aggregated feature vector of any sub-word sequence is non-autoregressive decoded to obtain the target positioning coordinate sequence; Alternatively, non-autoregressive decoding is performed on the aggregated feature vector of any sub-word sequence to obtain the initial positioning coordinate sequence corresponding to any sub-word sequence; non-autoregressive decoding based on an attention mechanism is performed on the image features, the aggregated feature vector, and the initial coordinate sequence to obtain the target positioning coordinate sequence; the accuracy of the target coordinate sequence is higher than that of the initial positioning coordinate sequence.

5. The method according to claim 3 or 4, characterized in that, The text recognition method is implemented through a text recognition model, including: The image encoding module of the text recognition model extracts features from the image to be recognized to obtain image features; The word sequence is obtained by performing autoregressive decoding based on the image features through the autoregressive decoding module of the text recognition model; the autoregressive decoding module belongs to the large model. The deep features of any sub-word sequence are fused by the fusion module of the text recognition model to obtain the aggregated feature vector of any sub-word sequence; The text recognition model uses a non-autoregressive decoding module to perform non-autoregressive decoding based on target features to obtain the target location coordinate sequence; the non-autoregressive decoding module includes a lightweight non-autoregressive decoding module and an attention-based non-autoregressive decoding module.

6. The method according to claim 5, characterized in that, The text recognition model was trained in the following way: The autoregressive decoding module was pre-trained in a self-supervised manner by adding a plain text corpus with hierarchical tagging information. A pre-trained text detection and segmentation model is obtained by pre-training a first-class image sample labeled with text unit category, location information, and mask. The text detection and segmentation model includes: the image encoding module and the non-autoregressive decoding module based on the attention mechanism; each text unit belongs to a level. The lightweight non-autoregressive decoding module, the fusion module, the pre-trained image encoding module, the autoregressive decoding module, and the attention-based non-autoregressive decoding module are jointly trained using second-class image samples of text based on labeled text unit location information, masks, and hierarchical labeling information to obtain the trained text recognition model. The accuracy of the location coordinate sequence decoded by the attention-based non-autoregressive decoding module is higher than that of the location coordinate sequence decoded by the lightweight non-autoregressive decoding module.

7. The method according to claim 1, characterized in that, The text content includes text units from at least some of the following levels: Title level, paragraph level, illustration level, table level, text line level, character level; The text units at the title level include text units at the paragraph level, text units at the illustration level, and text units at the table level; text units at the paragraph level contain text units at the line level; text units at the illustration level contain text units at the line level; text units at the table level contain text units at the line level; and text units at the line level contain text units at the character level.

8. An electronic device, characterized in that, The electronic device includes at least one processor and a memory connected to the processor; wherein: The memory is used to store computer programs; The processor is used to execute the computer program to enable the electronic device to implement the text recognition method as described in any one of claims 1 to 7.

9. A computer program product, characterized in that, It includes computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the text recognition method as described in any one of claims 1 to 7.

10. A computer storage medium, characterized in that, The storage medium carries one or more computer programs that, when executed by an electronic device, enable the electronic device to implement the text recognition method as described in any one of claims 1 to 7.

Images