A mathematical expression image recognition method, device and storage medium

Abstract

The present invention discloses a method, device and storage medium for mathematical expression image recognition, which belongs to the field of image recognition technology. The method includes: obtaining image data, annotating the image data, and constructing a training set; preprocessing the data set; constructing a mathematical expression recognition model, and training the mathematical expression recognition model using the preprocessed training set; obtaining a mathematical expression image to be recognized, and inputting the trained mathematical expression recognition model into the trained mathematical expression recognition model to obtain a recognition result; wherein, a multi-task joint optimization strategy is used to train and optimize the mathematical expression recognition model so that the model has both autoregressive and semi-autoregressive reasoning capabilities. The present invention proposes a simple and effective joint optimization strategy, which makes up for the deficiency of the context dependence of the semi-autoregressive model and improves the performance of the autoregressive model, so as to achieve accurate and rapid recognition of different types of mathematical expressions.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to a method, device and storage medium for mathematical expression image recognition. Background Art

[0002] Mathematical expressions often contain unique and concise information in documents. Converting photographed or scanned mathematical expression images into editable typeset language for information retrieval, editing, and understanding has important applications in automated examination marking, document digitization, office automation, assisted academic writing, and other human-computer interaction scenarios. However, manually entering mathematical expressions often requires the inputter to have a high level of mastery of typesetting syntax and tools, and is often inefficient. Unlike conventional one-dimensional text, mathematical expressions have complex spatial relationships between symbols. These spatial position relationships represent different logical relationships between symbols. In addition to identifying symbols, mathematical expression recognition also requires the correct parsing of these relationships.

[0003] In recent years, with the development of deep neural networks, academia and industry have achieved remarkable results in deep learning-based optical character recognition (OCR) tasks, such as natural scene text recognition, license plate recognition, and ID card verification. However, research in mathematical expression recognition has progressed more slowly. Key technical difficulties include the complex relationships between symbols in expressions, making it difficult to extract structured output information; the complex output structure, long output sequences, and slow inference speed; and the wide range of handwritten styles, arbitrary layouts, low-resolution photographic images, and significant background interference. These technical difficulties severely hinder the recognition of mathematical expressions. Summary of the Invention

[0004] In order to solve at least one of the technical problems existing in the prior art to a certain extent, the present invention aims to provide a method, device and storage medium for mathematical expression image recognition.

[0005] The technical solution adopted in the present invention is:

[0006] A mathematical expression image recognition method comprises the following steps:

[0007] Obtain image data of a single line or multiple lines of mathematical expressions, annotate the image data at the expression block level, and construct a training set;

[0008] The single-line mathematical expression images and annotations are randomly spliced ​​into multi-line mathematical expression images, while the original multi-line mathematical expressions remain unchanged. All annotations are converted into a two-dimensional double-end structure to obtain the preprocessed training set.

[0009] Constructing a mathematical expression recognition model, and training the mathematical expression recognition model using the preprocessed training set;

[0010] Acquire a mathematical expression image to be recognized, and input the trained mathematical expression recognition model to obtain a recognition result;

[0011] Among them, a multi-task joint optimization strategy is adopted to train and optimize the mathematical expression recognition model: ordinary autoregression and semi-autoregression are used for task supervision at the same time, so that the mathematical expression recognition model has both autoregressive and semi-autoregressive reasoning capabilities.

[0012] Furthermore, the annotation content includes the classification category corresponding to a single character, the relationship between superscripts and subscripts and fraction lines between different characters, and the line break symbols between different lines;

[0013] The performing expression block-level annotation on the image data includes:

[0014] For multi-line mathematical expression text blocks, mark them in order from top to bottom;

[0015] For images with multiple text blocks, the order of annotations between the text blocks is determined by reading order.

[0016] Furthermore, the randomly splicing of the single-line mathematical expression image and the annotation into a multi-line mathematical expression image includes:

[0017] Randomly select different single-line mathematical expression images, scale them to the same width or fill in shorter expressions to synthesize multi-line mathematical expression images, and splice the corresponding text annotations with line break symbols.

[0018] Furthermore, converting all annotations into a two-dimensional double-ended structure includes:

[0019] The one-dimensional sequence annotation is split and arranged into a two-dimensional structure with row arrangement according to its line break;

[0020] Dividing the labels of each row in the two-dimensional structure with row arrangement into a left-to-right sequence and a right-to-left sequence according to the number of characters;

[0021] When the number of characters in the left-to-right sequence and the right-to-left sequence is not equal, a padding character is inserted to align the character lengths of the left-to-right sequence and the right-to-left sequence.

[0022] Furthermore, the mathematical expression recognition model includes: a CNN visual encoder, a feature converter, a row-blocking dual-end mask module, and a row-aware position encoding module;

[0023] The CNN visual encoder is used to extract visual features of the input image;

[0024] The feature converter is used to decode the image features extracted by the CNN visual encoder into text features;

[0025] The row-block double-end mask module is used to mask the ungenerated text in other rows during training and inference, thereby ensuring the semi-autoregressive property of the feature converter;

[0026] The row-aware position encoding module is used to encode the text features in terms of row position and bi-end sequence.

[0027] Furthermore, in the result reasoning, the feature converter outputs the results of the left-to-right sequence and the right-to-left sequence of all rows in parallel and simultaneously; the output result of the feature converter at the current moment is used to obtain the input sequence of the next converter through the row-aware displacement algorithm, and the row-aware position encoding module and the row block dual-end mask module are updated until all rows are decoded to obtain the final result.

[0028] Furthermore, in the result inference, the feature converter is used to perform inference according to the semi-autoregressive decoding method, and to perform inference using the autoregressive method according to actual needs.

[0029] Furthermore, in model training, the loss function used is any combination of ordinary one-dimensional sequence, two-dimensional row structure sequence and two-dimensional row structure double-ended sequence.

[0030] Another technical solution adopted in the present invention is:

[0031] A mathematical expression image recognition device, comprising:

[0032] at least one processor;

[0033] at least one memory for storing at least one program;

[0034] When the at least one program is executed by the at least one processor, the at least one processor implements the method described above.

[0035] Another technical solution adopted in the present invention is:

[0036] A computer-readable storage medium stores a program executable by a processor, wherein the program executable by the processor is used to perform the method described above when executed by the processor.

[0037] The beneficial effect of the present invention is that the present invention proposes a simple and effective joint optimization strategy, which makes up for the deficiency of context dependence of the semi-autoregressive model and improves the performance of the autoregressive model to achieve accurate and rapid recognition of different types of mathematical expressions. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following introduction is made to the drawings of the embodiments of the present invention or the related technical solutions in the prior art. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative work.

[0039] Figure 1 is a sample diagram of a multi-line mathematical expression data set collected in an embodiment of the present invention;

[0040] Figure 2 Schematic diagram of splicing and synthesizing a single-line mathematical expression in an embodiment of the present invention;

[0041] Figure 3 Schematic diagram of a mathematical expression recognition model constructed in an embodiment of the present invention;

[0042] Figure 4 is a visualization diagram of the row perception attention mechanism according to an embodiment of the present invention;

[0043] Figure 5 Schematic diagram of a row block mask according to an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention and are not to be construed as limiting the present invention. The step numbers in the following embodiments are provided for ease of explanation only and do not limit the order of the steps. The order of execution of the steps in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0045] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, they cannot be understood as limitations on the present invention.

[0046] In the description of the present invention, "several" means one or more, "many" means more than two, "greater than," "less than," and "exceed" are understood to exclude the number itself, while "above," "below," and "within" are understood to include the number itself. The use of "first" and "second" in the description is solely for the purpose of distinguishing technical features and should not be construed as indicating or implying relative importance, implicitly specifying the number of the indicated technical features, or implicitly specifying the order of the indicated technical features.

[0047] Furthermore, in the description of this invention, unless otherwise specified, "plurality" refers to two or more. "And / or" describes the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0048] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, and connecting should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.

[0049] To address existing technical issues, this embodiment provides a mathematical expression recognition method based on semi-autoregressive decoding, which improves the transcription efficiency of mathematical expression images, saves manpower and material resources, and achieves efficient automatic recognition of mathematical expressions. The method specifically includes the following steps:

[0050] S1. Obtain data: collect image data of mathematical expressions, annotate the image data at the expression block level, and obtain a constructed training set and test set.

[0051] As an optional implementation, the expression block-level annotation content obtained in the data includes the classification category corresponding to a single character, the relationship between superscripts and subscripts and fraction lines between different characters, and line breaks between different lines; multi-line expression text blocks are annotated in order from top to bottom; for images with multiple text blocks, the annotation order between text blocks is determined according to their reading order.

[0052] S2. Preprocess data: Randomly combine the collected single-line expression images and annotations into multi-line expression images, keep the multi-line mathematical expressions unchanged, and convert all annotations into a two-dimensional double-end structure according to the row structure.

[0053] As an optional implementation, different single-line mathematical expression images are randomly selected from the preprocessed data, scaled to the same pixel width and filled to synthesize multi-line mathematical expression images, and the corresponding text annotations are spliced ​​using line break symbols.

[0054] As an optional implementation, the two-dimensional double-ended structure in the preprocessed data is to divide and arrange the one-dimensional sequence annotations into a two-dimensional structure with row arrangement according to its row structure; at the same time, the annotations of each row in the two-dimensional structure with row arrangement are divided into a left-to-right sequence and a right-to-left sequence according to the number of characters; when the number of characters in the left-to-right sequence and the right-to-left sequence are not equal, a padding character is inserted to align the character lengths of the left-to-right sequence and the right-to-left sequence.

[0055] S3. Model training: constructing a mathematical expression recognition framework based on the image feature extraction network and the transformer network, and training the preprocessed training data set under the mathematical expression recognition framework to obtain a mathematical expression recognition model.

[0056] As an optional implementation, the mathematical expression recognition model includes: a CNN visual encoder, a feature converter, a row block dual-end mask module, and a row-aware position encoding module;

[0057] The CNN visual encoder is used to extract visual features of the input image;

[0058] The feature converter is used to decode the image features extracted by the CNN visual encoder into text features;

[0059] The row-block double-ended mask module is used to mask text that is not generated during training and inference, thereby ensuring the semi-autoregressive nature of the feature converter;

[0060] The row-aware position encoding module is used to encode the text features sent to the feature converter in terms of row position and bi-directional sequence.

[0061] Specifically, a multi-task joint optimization strategy is adopted to train and optimize the mathematical expression recognition model: ordinary autoregression and semi-autoregression are simultaneously used for task supervision, so that the mathematical expression recognition model has both autoregressive and semi-autoregressive reasoning capabilities.

[0062] As an optional implementation, the loss function used in model training is any combination of ordinary one-dimensional sequence, two-dimensional row structure sequence and two-dimensional row structure double-ended sequence.

[0063] As an optional implementation, during result inference, the converter outputs the results of all rows from left to right and from right to left in parallel. The output result of the converter at the current moment is used to obtain the input sequence of the next converter through the row-aware shift algorithm, and the row-aware position encoding module and the row block dual-end mask module are updated until all rows are decoded to obtain the final result.

[0064] As an optional implementation, in result inference, the feature converter is used to perform inference according to the semi-autoregressive decoding method, and to perform inference using an autoregressive method according to actual needs.

[0065] S4. Result reasoning: For the image to be recognized, it is fed into the trained mathematical expression recognition model to obtain the recognized mathematical expression result.

[0066] The above method is explained in detail below with reference to the accompanying drawings and specific embodiments.

[0067] like Figure 3 As shown, this embodiment provides a mathematical expression recognition method based on semi-autoregressive decoding, comprising the following steps:

[0068] Step 1: Construct training and test sets based on images and annotations of single-line and multi-line mathematical expressions;

[0069] Step 2: Randomly splice the collected single-line expression images and annotations into multi-line expression images, and convert all annotations into a two-dimensional double-end structure;

[0070] Step 3: Build a mathematical expression recognition model based on the CNN image feature encoder and transformer network;

[0071] Step 4: Recognize the test set images based on the trained mathematical expression recognition model to obtain the annotations corresponding to the mathematical expressions.

[0072] In this embodiment, first, a training data set and a test data set for mathematical expression recognition are constructed. The data set of this embodiment contains a total of 89,964 mathematical expression images, from which 79,979 mathematical expression images are randomly sampled as the training data set, and the remaining 9,985 mathematical expression images are used as the test data set. Figure 1 shown; Figure 1 (a), (b), (c), (d), (e), and (f) are examples of different expression images. The collected multi-line mathematical expression dataset contains a total of approximately 4.2 million characters, with an average number of characters of 47, a maximum number of lines of 45, and an average number of lines of 4.24. Specific data statistics are shown in Table 1.

[0073] Table 1. Distribution statistics of the collected multi-line mathematical expression dataset

[0074] Data Type Printing + Handwriting The amount of training set data 79979 The number of test set data 9985 Symbol category 457 Total characters 4.2 million Maximum character length 345 Average character length 47.0 Maximum number of rows 45 Average number of rows 4.24

[0075] For a single-line expression, follow the Figure 2The images of mathematical expressions in the test set are randomly spliced ​​in the manner shown in the figure to obtain synthetic multi-line mathematical expression images and annotations, which are then added to the training set. Among them, the mathematical expression images in the test set do not appear in the training set.

[0076] In the data preprocessing stage, this embodiment divides the collected annotations containing multiple lines of mathematical expressions into rows. In this embodiment, a mathematical expression sequence is defined, where n lines where k i Denotes the number of characters in the i-th row. Row-aware decoding decodes a character from all rows simultaneously at the current time. At time step t, the decoder predicts the character sequence Instead of autoregressive decoding of individual characters, for each row i, the dual-end decoding strategy predicts the left-to-right and right-to-left characters at time step t and Combined with the above line-by-line decoding strategy, the line-aware dual-end decoding strategy predicts the character sequence from left to right at time step t and right-to-left character sequences By arranging these two sequences in time sequence, a two-dimensional double-ended structure can be obtained, e.g. Figure 3 As shown in .

[0077] During the model training phase, the acquired multi-line mathematical expression images are used as input, and the two-dimensional double-end structure is converted to one dimension as the target sequence. During the training process, all images in the training dataset are scaled to a fixed pixel size of 256×256 and filled with white edges. Figure 3 As shown, the row-aware semi-autoregressive decoding transformer in this embodiment follows the encoder-decoder framework. The CNN visual feature extraction network encodes the visual features of the input image. The visual features encoded with 2D position information are then fed into the Transformer decoder as interactive attention as a query vector. The row-aware self-attention mechanism applies a row block mask to the attention map to eliminate the weight of characters that have not yet been generated in the attention mechanism. The row-aware position encoding gives the model row order perception and directionality through a learnable row index vector and a mutual inverse sine position encoding. During training, the model is supervised by cross-entropy loss using the one-dimensional sequence of the two-dimensional double-end structure and the original one-dimensional sequence.

[0078] 1) CNN Encoder

[0079] The CNN encoder uses a 16x downsampled DenseNet with a growth rate of 24 and 16 layers per block. The specific structure is shown in Table 2:

[0080] Table 2

[0081] Network layer Specific operations Feature map size Original image enter 3×256×256 Input layer 7*7 convolution + ReLU + 2x maximum pooling 48×64×64 dense layer 16 bottleneck layers 432×64×64 Transformation Layer Batch Normalization + Convolution + Random Dropout 216×32×32 dense layer 16 bottleneck layers 600×32×32 Transformation Layer Batch Normalization + Convolution + Random Dropout 300×16×16 dense layer 16 bottleneck layers 684×16×16 Output layer 1*1 convolution + ReLU + LayerNorm 16×16×256

[0082] The output of each dense layer is the concatenation of the calculation result of the layer and the input.

[0083] 2) Transformer Decoder

[0084] The transformer uses a Transformer decoder structure. The Transformer decoder model used in this embodiment has a dimension D of 256, the number of multi-head attention is 8, the number of decoding layers is 3, the dimension of the feedforward network is 1024, and the maximum number of rows is set to 16.

[0085] 3) Row-aware self-attention mechanism

[0086] In order to implement the row-aware dual-end decoding strategy, this embodiment proposes a mask with row-block dual-end mask to adjust the regression relationship between different characters. In row-aware decoding, this embodiment performs autoregressive decoding on different rows in parallel. For each row i, the mask is a triangular matrix. Figure 5 As shown, the row block mask proposed in this embodiment The triangular matrix of each row is connected diagonally to achieve parallelism. The proposed double-ended decoding requires left-to-right and right-to-left decoding. In order to achieve a single row of right-to-left autoregression without changing the sequence order, this embodiment changes the lower triangular matrix to an upper triangular matrix, such as Figure 5 (b) shown.

[0087] The double-ended mask is a combination of left-to-right and right-to-left autoregressive masks, such as Figure 5 (c) and (d) yield Figure 5 (e) is shown in the steps.

[0088] The final row-block dual-end mask combines the row-block mask and the dual-end mask. In order to better interact with the semantic information of other rows, this embodiment allows cross-row attention, such as Figure 5 As shown in the rounded rectangle in (e).

[0089] 4) Row-aware position encoding

[0090] In order to encode the position of the current decoded line text, this embodiment proposes a learnable line index encoding vector:

[0091] P l (i) = Emb(W i |i;W)

[0092] in is a learnable parameter, i is the row index to be encoded, and is the row index encoding vector obtained by querying the embedding index i in W.

[0093] In dual-end decoding, the perception of the decoding direction is crucial. Therefore, this embodiment proposes a reciprocal inverse sine encoding:

[0094]

[0095]

[0096] The above formulas represent the position encoding from left to right and from right to left respectively, where k represents the number of characters in the current line, x∈[0,k i / 2), D is the embedding dimension, and T is a temperature constant set to 1000.

[0097] Finally, to obtain the final row-aware bi-terminal positional encoding, this embodiment assigns the same row index encoding vector to all characters in the same row, and then adds two inverse sine codes:

[0098]

[0099] 5) Joint Optimization

[0100] This example proposes a simple yet effective joint optimization strategy for semi-autoregressive tasks. This strategy overcomes the context-dependency limitations of semi-autoregressive models while improving their performance. Specifically, multiple tasks are optimized simultaneously. This example utilizes left-to-right and right-to-left prediction tasks, as well as the row-aware two-ended semi-autoregressive task. We simultaneously optimize parameters by summing the cross-entropy losses of the three tasks for each batch of samples.

[0101]

[0102]

[0103]

[0104]

[0105] Where N and N ′ are the lengths of the autoregressive and semi-autoregressive sequences, respectively, and λ is a coefficient constant.

[0106] In the result stage, this embodiment will add the predicted characters to the input sequence by bidirectionally shifting the output of each row during the inference process. The model implements the semi-autoregressive decoding, as shown in Figure 4 To implement this reasoning method, this embodiment designs a pseudo code such as Algorithm 1, as shown in Table 3 below:

[0107] Table 3

[0108]

[0109] The one-dimensionalization of a two-dimensional double-terminal structure represents arranging the two-dimensional double-terminal structure in a certain order to form a one-dimensional sequence. It is worth noting that the one-dimensionalization method in this embodiment is only a special case, and similar transformations and their corresponding block matrix masks are also included in the scope of protection of this application.

[0110] In summary, the embodiments of the present invention analyze the shortcomings of existing deep learning methods and propose new ideas for mathematical expression recognition. The new ideas mainly include using deep learning networks to extract features from images, using transformer networks to convert image features into text features, and realizing semi-autoregressive output at the row level, thereby enhancing the model's perception of the spatial structure of expressions. The new ideas have the advantages of robust row structure recognition, fast inference speed, and accurate recognition results.

[0111] This embodiment further provides a mathematical expression image recognition device, comprising:

[0112] at least one processor;

[0113] at least one memory for storing at least one program;

[0114] When the at least one program is executed by the at least one processor, the at least one processor implements the following Figure 3 The method shown.

[0115] A mathematical expression image recognition device of this embodiment can execute a mathematical expression image recognition method provided by the method embodiment of the present invention, can execute any combination of implementation steps of the method embodiment, and has the corresponding functions and beneficial effects of the method.

[0116] The present application also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs Figure 3 The method shown.

[0117] This embodiment also provides a storage medium that stores instructions or programs that can execute a mathematical expression image recognition method provided by an embodiment of the method of the present invention. When the instructions or program are run, any combination of implementation steps of the method embodiment can be executed, and the corresponding functions and beneficial effects of the method can be obtained.

[0118] In some optional embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided in an exemplary manner for the purpose of providing a more comprehensive understanding of the technology. The disclosed method is not limited to the operation and logic flow presented herein. Optional embodiments are contemplated in which the order of the various operations is changed and the sub-operations described as a part of a larger operation are performed independently.

[0119] Furthermore, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise indicated, one or more of the functions and / or features described may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It will also be understood that a detailed discussion of the actual implementation of each module is not necessary for understanding the present invention. More specifically, given the properties, functions, and internal relationships of the various functional modules in the devices disclosed herein, the actual implementation of the module will be understood within the ordinary skill of an engineer. Therefore, a person skilled in the art using ordinary skill will be able to implement the present invention set forth in the claims without undue experimentation. It will also be understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0120] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0121] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0122] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0123] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0124] In the above description of this specification, reference to the terms "one embodiment / example," "another embodiment / example," or "certain embodiments / examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0125] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

[0126] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A mathematical expression image recognition method, characterized in that: The following steps are involved: Obtain image data of a single line or multiple lines of mathematical expressions, annotate the image data at the expression block level, and construct a training set; The single-line mathematical expression images and annotations are randomly spliced ​​into multi-line mathematical expression images, while the original multi-line mathematical expressions remain unchanged. All annotations are converted into a two-dimensional double-end structure to obtain the preprocessed training set. Constructing a mathematical expression recognition model, and training the mathematical expression recognition model using the preprocessed training set; Acquire a mathematical expression image to be recognized, and input the trained mathematical expression recognition model to obtain a recognition result; The mathematical expression recognition model is trained and optimized by adopting a multi-task joint optimization strategy: ordinary autoregression and semi-autoregression are simultaneously used for task supervision, so that the mathematical expression recognition model has both autoregressive and semi-autoregressive reasoning capabilities; The mathematical expression recognition model includes: a CNN visual encoder, a feature converter, a row block dual-end mask module, and a row perception position encoding module; The CNN visual encoder is used to extract visual features of the input image; The feature converter is used to decode the image features extracted by the CNN visual encoder into text features; The row-block double-ended mask module is used to mask text that is not generated during training and inference, thereby ensuring the semi-autoregressive nature of the feature converter; The row-aware position encoding module is used to encode the text features in terms of row position and bi-end sequence.

2. A mathematical expression image recognition method according to claim 1, characterized in that: The annotation content includes the classification category corresponding to a single character, the relationship between superscripts and subscripts and fraction lines between different characters, and the line break symbols between different lines; The performing expression block-level annotation on the image data includes: For multi-line mathematical expression text blocks, mark them in order from top to bottom; For images with multiple text blocks, the order of annotations between the text blocks is determined by reading order.

3. A mathematical expression image recognition method according to claim 1, characterized in that: The randomly splicing of the single-line mathematical expression image and the annotation into a multi-line mathematical expression image includes: Randomly select different single-line mathematical expression images, scale them to the same width or fill in shorter expressions to synthesize multi-line mathematical expression images, and splice the corresponding text annotations with line break symbols.

4. A mathematical expression image recognition method according to claim 1, characterized in that: The conversion of all annotations into a two-dimensional double-ended structure includes: The one-dimensional sequence annotation is split and arranged into a two-dimensional structure with row arrangement according to its line break; Dividing the labels of each row in the two-dimensional structure with row arrangement into a left-to-right sequence and a right-to-left sequence according to the number of characters; When the number of characters in the left-to-right sequence and the right-to-left sequence is not equal, a padding character is inserted to align the character lengths of the left-to-right sequence and the right-to-left sequence.

5. The method for mathematical expression image recognition according to claim 1, wherein: In result inference, the feature converter outputs the results of the left-to-right sequence and the right-to-left sequence of all rows in parallel and simultaneously; the output result of the feature converter at the current moment is used to obtain the input sequence of the next converter through the row-aware displacement algorithm, and the row-aware position encoding module and the row block dual-end mask module are updated until all rows are decoded to obtain the final result.

6. A mathematical expression image recognition method according to claim 1, characterized in that: In result inference, the feature converter is used to perform inference according to the semi-autoregressive decoding method, and to perform inference using the autoregressive method according to actual needs.

7. A mathematical expression image recognition method according to claim 1, characterized in that: During model training, the loss function used is any combination of ordinary one-dimensional sequences, two-dimensional row-structured sequences, and two-dimensional row-structured double-ended sequences.

8. A mathematical expression image recognition device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 7.

9. A computer-readable storage medium storing a program executable by a processor, characterized in that: The processor-executable program is used to perform the method according to any one of claims 1 to 7 when executed by the processor.

Images