Handwritten mathematical formula identification method and system

Abstract

The invention belongs to the technical field of information technology and image recognition, and relates to a handwritten mathematical formula recognition method and system. The method comprises the following steps: constructing handwritten mathematical formula annotation data; constructing an initial handwritten mathematical formula recognition model, training the initial model by using the annotation data, and performing joint optimization on a sequence decoding task and a tree structure prediction task to obtain a handwritten mathematical formula recognition model; and carrying out handwritten mathematical formula identification by adopting the handwritten mathematical formula identification model obtained by training. According to the method, the advantages of a sequence decoding model and a tree decoding model are fused, the understanding and generalization ability of the model to a mathematical formula structure are effectively improved, and the accuracy of handwritten mathematical formula recognition is remarkably improved on the basis of keeping the efficient parallel training advantages of an existing encoder-decoder model-based method.

Description

Technical Field

[0001] The invention belongs to the fields of information technology and image recognition technology, and specifically relates to a handwritten mathematical formula recognition method and system. Background Art

[0002] Handwritten mathematical formula recognition can identify and parse mathematical formulas from handwritten materials, and has important applications in many fields such as education, scientific research, engineering design, and data analysis. This technology can help us quickly and accurately convert and process complex mathematical formulas, reduce repetitive work, and provide convenience for people's lives and work. For example, in the field of education, handwritten formula recognition can help teachers and students quickly convert formulas in blackboards or notes into electronic documents, improving teaching and learning efficiency. In the field of scientific research, researchers can use this technology to quickly extract and analyze mathematical formulas from manuscripts and accelerate the scientific research process. In addition, handwritten formula recognition can also help engineers and data analysts quickly convert and apply various complex formulas in design and data processing, improving work efficiency and accuracy. By efficiently recognizing and digitizing handwritten formulas, users can better integrate and utilize this information to promote the development of related fields.

[0003] Handwritten mathematical formula recognition is divided into online recognition and offline recognition. The difference between the two is that the input method is different. Online input is dynamic in nature, while offline input is static. Online input is the handwritten stroke trajectory, which includes the sequence and curvature of the strokes; offline input is an image containing a handwritten mathematical formula. The recognition method of the present invention belongs to the offline handwritten mathematical formula recognition method.

[0004] Mathematical formulas are composed of symbols and symbol relationships. Symbols include Arabic numerals, English letters, Greek letters, operators and other special characters. Symbol relationships are different from conventional linear character sequence relationships. They present multi-level and highly structured characteristics. For example, mathematical symbols such as fractions, square roots, integrals, etc. not only contain basic characters, but also involve complex structures such as superscripts and subscripts, nested symbols, etc., which greatly increase the difficulty of recognition. Therefore, in the task of recognizing handwritten mathematical formulas, the core challenge is how to effectively model the symbols and structural relationships of mathematical formulas. This not only requires the recognition method to accurately identify the characters in the image, but more importantly, to construct complex hierarchical relationships between these characters. In addition, due to the diversity of handwriting, the similarity of handwritten symbols, the differences in handwriting styles, the complex structure of mathematical formulas and contextual dependencies, the recognition difficulty is greatly increased.

[0005] In the field of handwritten mathematical formula recognition technology, early researchers used rule-based methods. With the continuous advancement of deep learning technology, the encoder-decoder based recognition method can closely combine the input image and the recognition content, so that the model can fully understand the content in the image and transcribe it into an accurate character sequence based on the image content. It has strong modeling capabilities and shows superior performance compared to traditional methods.

[0006] In computer systems, the representation methods of mathematical formulas are mainly divided into two types: sequence-based representation and tree-based representation. The encoder-decoder-based recognition method in handwritten mathematical formula recognition has also developed two corresponding decoding models: a sequence-based decoding model and a tree-based decoding model. The method based on the sequence decoding model uses LaTeX expression as the decoding target. Its advantage is that LaTeX, as a widely used mathematical formula expression method, can not only adapt well to existing research results in fields such as natural language processing and optical character recognition, but also has strong versatility. The method based on the tree decoding model regards mathematical formulas as tree structures for predictive decoding. Its advantage is that the model design fully incorporates the characteristics of the tree structure, which theoretically enhances the model's ability to understand mathematical formulas and ensures that the model can always decode legal mathematical formulas during the predictive reasoning process.

[0007] The method based on the sequence decoding model has certain limitations. First, the tree structure characteristics of mathematical formulas are not fully considered during the training process, which may lead to insufficient understanding of the formula structure, thus affecting its generalization ability when dealing with complex structure formulas. Second, as the target of sequence prediction, LaTeX expression cannot guarantee the grammatical norms of the model decoding results. For example, when dealing with complex mathematical formulas, the model may produce unmatched left and right curly brackets, resulting in prediction results that do not conform to the LaTeX grammatical norms.

[0008] At the same time, previous methods based on tree-based decoding models also have shortcomings. First, these models need to rely on recurrent neural networks for construction and cannot be efficiently trained in parallel. In addition, the decoding methods of these models are relatively complex and lack versatility and wide applicability compared to LaTeX expressions; moreover, their performance is usually inferior to methods based on sequence decoding models, resulting in the failure of tree-based decoding methods to be widely used. Summary of the invention

[0009] In view of the respective advantages and disadvantages of the above two decoding models, the present invention combines the advantages of the sequence decoding model and the tree decoding model, and proposes a handwritten mathematical formula recognition method and system based on formula tree structure perception.

[0010] The technical solution adopted by the present invention is as follows:

[0011] A method for recognizing handwritten mathematical formulas comprises the following steps:

[0012] Construct handwritten mathematical formula annotation data;

[0013] Construct an initial model for handwritten mathematical formula recognition, use the labeled data to train the initial model, and perform joint optimization on the sequence decoding task and the tree structure prediction task to obtain a handwritten mathematical formula recognition model;

[0014] The trained handwritten mathematical formula recognition model is used to perform handwritten mathematical formula recognition.

[0015] Furthermore, the constructing of handwritten mathematical formula annotation data includes converting the mathematical formula into a tree structure; the converting of the mathematical formula into a tree structure includes: representing the tree structure of the mathematical formula as a series of tuples, wherein each tuple (c, p) contains the child node c of each subtree and its parent node p.

[0016] Furthermore, the model structure of the initial model includes:

[0017] A visual encoding model that extracts high-level visual features from the input handwritten math formula slices;

[0018] The image position encoding module and the character position encoding module are used to provide explicit position information for visual features and word embedding vectors respectively;

[0019] A decoding model, used to output a sequence of character semantic features;

[0020] A formula tree structure perception module is used to construct the feature vector output by the decoding model into a formula structure tree;

[0021] The projection layer is used to decode the mathematical formula sequence in an autoregressive manner using the feature vector output by the decoding model.

[0022] Furthermore, the decoding model adopts a Transformer decoding model that introduces an overlay attention mechanism.

[0023] Furthermore, the processing process of the formula tree structure perception module includes:

[0024] Input the character semantic feature sequence X extracted by the decoding model;

[0025] Use the Transformer encoder to extract the character semantic feature sequence X to obtain X′;

[0026] Through two linear projection functions F c (·) and F p(·), map the feature vector X′ output by the Transformer encoder into child node vectors and parent node vectors respectively;

[0027] The child node vectors and parent node vectors are combined and added in pairs to obtain the relationship feature matrix M. The vector M at each position in the matrix M is i,j The feature vector representing the relationship between the character with subscript i as the child node and the character with subscript j as the parent node;

[0028] Use function F score (·) to convert the relationship feature matrix M into the relationship score matrix S, function F score (·) consists of a ReLU activation function and a vector dot product operation;

[0029] Each element S in the relation score matrix S output by the formula tree structure perception module i,j represents the relationship score with the i-th character as the child node and the j-th character as the parent node, where the higher the score, the greater the possibility of the parent-child relationship. For the element S with the highest score in each row, a,b , it is believed that the model predicts the existence of a parent-child relationship with the ath symbol as the child node and the bth symbol as the parent node, and then constructs the entire mathematical formula tree structure by establishing the relationship between the characters.

[0030] Furthermore, the training strategy of the initial model includes: combining the loss function of the sequence decoding task and the loss function of the tree structure prediction task to train the model;

[0031] The loss function of the sequence decoding task is:

[0032]

[0033] Among them, X t represents the feature vector of the t-th time step output by the decoding model, W o represents the linear projection parameter matrix, b o represents the bias vector;

[0034] The loss function of the tree structure prediction task is:

[0035]

[0036] Among them, S is the relationship score matrix output by the formula tree structure perception module, which is used to estimate the probability that each character t has a parent-child relationship with other characters.

[0037] Furthermore, the handwritten mathematical formula recognition model obtained by training is used to perform handwritten mathematical formula recognition, including:

[0038] Obtain a handwritten mathematical formula image to be recognized;

[0039] Preprocess the handwritten mathematical formula image to be recognized;

[0040] The preprocessed handwritten mathematical formula image is input into the trained handwritten mathematical formula recognition model, and the model predicts each symbol in the formula image and generates an initial recognition sequence;

[0041] A beam search strategy is used for the initial recognition sequence to generate a set of candidate sequences, and a sequence decoding score is calculated for each candidate sequence;

[0042] The formula tree structure perception module is used to calculate the relationship score matrix of the candidate sequence, and then the tree structure prediction score is calculated for each candidate sequence;

[0043] The sequence decoding score is added to the tree structure prediction score, and the candidate sequence with the highest comprehensive score is selected as the final output sequence.

[0044] A handwritten mathematical formula recognition system, comprising:

[0045] Annotation data construction module, used to construct annotation data of handwritten mathematical formulas;

[0046] The model training module is used to build an initial model for handwritten mathematical formula recognition. The initial model is trained using labeled data, and joint optimization is performed on the sequence decoding task and the tree structure prediction task to obtain a handwritten mathematical formula recognition model.

[0047] The formula recognition module is used to recognize handwritten mathematical formulas using the trained handwritten mathematical formula recognition model.

[0048] The beneficial effects of the present invention are as follows:

[0049] The present invention effectively improves the model's understanding of the structure of mathematical formulas and its generalization ability by integrating methods based on sequence decoding and tree decoding. On the basis of maintaining the advantages of efficient parallel training of the existing encoder-decoder model method, the accuracy of handwritten mathematical formula recognition is significantly improved. Specifically, the present invention can jointly optimize these two tasks in the training stage, and at the same time train the model's understanding of the sequence arrangement and tree structure of mathematical formulas. In the inference and prediction stage, the present invention proposes to integrate the tree structure prediction scoring mechanism into the beam search algorithm, so that the model considers both the sequence decoding of mathematical formulas and the rationality of their tree structure when generating LaTeX expressions, further improving the accuracy of recognition. In summary, the recognition method proposed by the present invention improves the recognition ability of the sequence decoding model for complex structure handwritten mathematical formulas. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a schematic diagram of the model structure of the present invention.

[0051] Figure 2 It is a schematic diagram of the formula tree structure perception module.

[0052] Figure 3 This is a sample image of a handwritten formula to be recognized. DETAILED DESCRIPTION

[0053] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below through specific embodiments and drawings.

[0054] The present invention combines the advantages of the sequence decoding model and the tree decoding model, and proposes a handwritten mathematical formula recognition method based on formula tree structure perception. On the basis of the flexibility and high training efficiency of the sequence decoding model, the formula tree structure perception module is introduced to improve the understanding and generalization ability of complex formula structures. By jointly optimizing the two tasks of sequence prediction and tree structure prediction, the present invention can consider the rationality of its formula tree structure while decoding the LaTeX sequence, so that the recognition result is more complete and accurate in the grammatical structure. The experimental results on multiple data sets show that the present invention exhibits better recognition performance than the traditional sequence decoding model and tree decoding model, and also illustrates the potential of the method combining sequence and tree decoding in the field of handwritten mathematical formula recognition.

[0055] The present invention is a handwritten mathematical formula recognition method based on an encoder-decoder, and the recognition process is completed by constructing a deep neural network model. First, handwritten mathematical formula annotation data is constructed, and an initial model for handwritten mathematical formula recognition is constructed. The initial model is trained using the annotation data, and the visual features of the handwritten mathematical formula and the semantic structure of the LaTeX expression are automatically learned from the data to obtain the final model. Then, the final model obtained by training is used to recognize handwritten mathematical formulas.

[0056] Step 1: Construct handwritten mathematical formula annotation data and convert the mathematical formula based on LaTeX expression into a tree structure.

[0057] Specifically, the present invention represents the tree structure of mathematical formulas as a series of pairs, where each pair (c, p) contains the child node c and the parent node p of each subtree. It is worth noting that in order to eliminate the ambiguity that may be introduced by repeated symbols and ensure compatibility with the LaTeX expression annotation used by the sequence decoding model, the present invention chooses to use the index of the mathematical symbol in the LaTeX expression as the node identifier in the formula tree instead of directly using the mathematical symbol itself. For example, for Figure 1 Mathematical formula in 3 2-1=8, its corresponding LaTeX expression is annotated as "3^{2}-1=8", and its formula tree structure annotation can be expressed as "(0,-1),(1,-1),(2,-1),(3,0),(4,-1),(5,0),(6,5),(7,6),(8,7)", where -1 means that the node has no parent node, that is, the node does not participate in the tree structure construction process. In this way, the present invention unifies the sequence annotation and tree structure annotation data of mathematical formulas, thereby allowing the model to perform end-to-end joint optimization on sequence decoding and tree structure prediction tasks.

[0058] Step 2: Build an initial model for handwritten mathematical formula recognition.

[0059] The model structure of the present invention is as follows Figure 1 As shown, it is mainly composed of the following four parts: 1) A visual encoding model, which is used to extract high-level visual features with rich meanings from the input handwritten mathematical formula pieces. This module adopts the DenseNet model widely used in the field of handwritten mathematical formula recognition as the backbone network. 2) An image position encoding module and a character position encoding module, which provide explicit position information for visual features and word embedding vectors respectively. The image position encoding module and the character position encoding module use the same sine and cosine position encoding. 3) A decoding model, which is used to output a sequence of character semantic features. The present invention adopts a Transformer decoding model that introduces a covering attention mechanism. 4) A formula tree structure perception module, which is used to construct the feature vector output by the decoding model into a formula structure tree, helping the model to better capture the structural information of mathematical formulas. 5) A projection layer, which is used to decode the LaTeX sequence in the form of autoregression using the feature vector output by the decoding model.

[0060] Among them, the visual encoding model can be implemented using a DenseNet model or the like.

[0061] The decoding model is as follows: Figure 1 As shown in the figure, it includes a self-attention module, an overlay attention module, and a feedforward neural network. The self-attention module is responsible for the correlation between the positions in the decoding sequence to model the information interaction between the current decoding position and the previous position. The overlay attention module is responsible for capturing the information interaction between the decoder and the encoder, guiding the decoder to generate the final output sequence using the feature information generated by the encoder. The feedforward neural network is responsible for nonlinear transformation of the representation after attention calculation, enhancing the expression ability of the model and generating more accurate feature representation.

[0062] The architecture overview of the formula tree structure perception module proposed in the present invention is as follows: Figure 2 As shown in the figure, the “SOS” mark is used to indicate the beginning of the sequence. The input of this module is the character semantic feature sequence X∈R extracted by the decoding model. T×d, where T represents the length of the LaTeX sequence and d represents the dimension of the character feature vector. In order to map these character semantic features into the semantic space of the formula tree structure, the present invention first uses a Transformer encoder to extract features from the character semantic feature sequence X to obtain X′∈R T×d , which facilitates the subsequent construction of a tree structure relationship in the semantic space.

[0063] X′=TransformerEncoder(X),

[0064] Then, through two linear projection functions F c (·) and F p (·), the feature vector X′ output by the Transformer encoder is mapped to the child node vector X c ∈R T×d and parent node vector X p ∈R T×d .

[0065] X c =F c (X′)=X′W c ,

[0066] X p =F p (X′)=X′W p ,

[0067] Where W c ∈R d×d and W p ∈R d×d are all trainable linear projection parameter matrices, where d represents the feature vector dimension of the character. In order to construct the formula tree structure relationship between characters, the present invention combines the child node vectors and the parent node vectors in pairs and adds them together to obtain the relationship feature matrix M. In this matrix, the vector M at each position i,j ∈R d They all represent the relationship feature vectors between the character with subscript i as the child node and the character with subscript j as the parent node in the LaTeX expression.

[0068]

[0069] Finally, the present invention uses the function F score (·) to convert the relation feature matrix M into the relation score matrix S∈R T ×T , which consists of a ReLU activation function and a vector dot product operation:

[0070] S=F score (M)=max(0,M)vs .

[0071] Among them, v s is the weight vector.

[0072] Each element S in the relation score matrix S output by the formula tree structure perception module i,j Both represent the relationship score with the i-th character as the child node and the j-th character as the parent node, where the higher the score, the greater the possibility of the parent-child relationship. In the mathematical formula tree structure, each node except the root node has only one parent node. Therefore, for the element S with the highest score in each row a,b , it can be considered that the model predicts the existence of a parent-child relationship with the a-th symbol as the child node and the b-th symbol as the parent node, and then constructs the entire mathematical formula tree structure by establishing the relationship between the characters.

[0073] Step 3: Use the labeled data to train the initial model to obtain the final model.

[0074] 1) Training strategy

[0075] In order to combine the sequence decoding and tree decoding tasks, and thereby enable the model to perform end-to-end joint optimization on these two tasks, the present invention combines the sequence decoding and tree structure prediction loss functions to train the model.

[0076] Specifically, for the LaTeX expression sequence y1,…,y T , in the sequence decoding task, the feature vector X∈R output by the decoding model T×L It can be used to calculate the probability of each character appearing at time step t, and then the cross entropy loss function is used to calculate the loss L of the sequence decoding task seq .

[0077]

[0078] Among them, X t represents the feature vector of the t-th time step output by the decoding model, W o represents the linear projection parameter matrix, b o Represents the bias vector.

[0079] In the tree structure prediction task, the relation score matrix S∈R output by the formula tree structure perception module is T×T , which is used to estimate the probability that each character t has a parent-child relationship with other characters. Then the cross entropy loss function is also used to calculate the loss L of the tree structure prediction task. struct .

[0080]

[0081] Finally, the losses of the sequence decoding and tree structure prediction tasks are added together to obtain the loss function L for training the method of the present invention.

[0082] L=L seq +L struct

[0083] 2) Model training

[0084] The present invention can use the PyTorch framework to build an initial model, use labeled data for supervised training, adjust the model optimization hyperparameters according to experimental conditions, and obtain the final model.

[0085] Step 4: Use the final model obtained through training to recognize handwritten mathematical formulas.

[0086] 1) Obtain the formula image to be recognized, and use scanning or mobile device to take a photo to obtain the handwritten formula image to be recognized. For example Figure 3 shown.

[0087] 2) Preprocess the handwritten formula to be recognized, perform image enhancement and noise removal on the formula image, and then perform tilt correction and scale normalization operations on the formula image, so that the formula image to be recognized has the same scale and angle as the images in the training image dataset.

[0088] 3) The preprocessed formula image is input into the final model obtained through training. The model predicts each symbol in the formula image and generates an initial recognition sequence.

[0089] 4) A beam search strategy is used for the initial recognition sequence to generate a set of candidate sequences, and the beam search scores of forward and reverse sequence decoding are used to calculate the sequence decoding score S for each candidate sequence y seq (y).

[0090] 5) Use the formula tree structure perception module to calculate the relationship score matrix of these candidate sequences, use the candidate sequence y to construct the corresponding tree structure annotation, calculate the cross entropy loss of the relationship score matrix and the tree structure annotation, and use the negative cross entropy loss to calculate the tree structure prediction score S for each candidate sequence struct (y).

[0091] 6) Add the sequence decoding score to the tree structure prediction score, that is, S seq (y)+S struct (y), select the candidate sequence with the highest comprehensive score as the final output sequence.

[0092] By considering the scores of sequence decoding and tree structure prediction at the same time, the above method not only ensures the accuracy of the final generated sequence, but also considers the rationality of its formula tree structure.

[0093] The key point of the present invention is the formula tree structure perception module. This module effectively improves the model's understanding of the structure of mathematical formulas and its generalization ability by integrating methods based on sequence decoding and tree decoding. Specifically, the module can jointly optimize these two tasks in the training phase, and at the same time train the model's understanding of the sequence arrangement and tree structure of mathematical formulas. In the inference prediction phase, the present invention proposes to integrate the tree structure prediction scoring mechanism into the beam search algorithm, so that the model considers both the sequence decoding of mathematical formulas and the rationality of their tree structure when generating LaTeX expressions, further improving the accuracy of recognition.

[0094] The above embodiment takes the mathematical formula expression in LaTeX format as an example to illustrate the handwritten mathematical formula recognition method of the present invention. In addition to LaTeX expression, the method of the present invention is also applicable to other forms of mathematical formula expressions, such as MathML.

[0095] The performance of the proposed model and the existing state-of-the-art models in formula recognition accuracy (ExpRate) were compared on the CROHME 2014 / 2016 / 2019 test datasets commonly used in handwritten math formula recognition tasks. All methods did not use data augmentation to ensure a fair comparison. As shown in Table 1, the proposed method consistently outperformed the existing methods in all indicators.

[0096] Table 1. Comparison of formula recognition accuracy between the present invention and the existing method

[0097]

[0098] Another embodiment of the present invention provides a handwritten mathematical formula recognition system, which includes:

[0099] Annotation data construction module, used to construct annotation data of handwritten mathematical formulas;

[0100] The model training module is used to build an initial model for handwritten mathematical formula recognition. The initial model is trained using labeled data, and joint optimization is performed on the sequence decoding task and the tree structure prediction task to obtain a handwritten mathematical formula recognition model.

[0101] The formula recognition module is used to recognize handwritten mathematical formulas using the trained handwritten mathematical formula recognition model.

[0102] The division of the above modules is only an example. In actual applications, the above functions can be assigned to different functional modules as needed to complete all or part of the functions described in the above method. The specific working process of each module can refer to the corresponding process in the above method embodiment, which will not be repeated here.

[0103] Another embodiment of the present invention provides a computer device (computer, server, smart phone, etc.), which includes a memory and a processor, the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program includes instructions for executing each step in the method of the present invention.

[0104] Another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, magnetic disk, optical disk), wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a computer, the steps of the method of the present invention are implemented.

[0105] The specific embodiments of the present invention disclosed above are intended to help understand the content of the present invention and implement it accordingly. It can be understood by those skilled in the art that various replacements, changes and modifications are possible without departing from the spirit and scope of the present invention. The present invention should not be limited to the contents disclosed in the embodiments of this specification, and the scope of protection of the present invention shall be subject to the scope defined in the claims.

Claims

1. A method for recognizing handwritten mathematical formulas, characterized in that: The following steps are involved: Construct handwritten mathematical formula annotation data; Construct an initial model for handwritten mathematical formula recognition, use the labeled data to train the initial model, and perform joint optimization on the sequence decoding task and the tree structure prediction task to obtain a handwritten mathematical formula recognition model; The trained handwritten mathematical formula recognition model is used to perform handwritten mathematical formula recognition.

2. The method according to claim 1, characterized in that The constructing of handwritten mathematical formula annotation data includes converting the mathematical formula into a tree structure; the converting of the mathematical formula into a tree structure includes: representing the tree structure of the mathematical formula as a series of tuples, wherein each tuple (c, p) contains the child node c of each subtree and its parent node p.

3. The method according to claim 1, characterized in that The model structure of the initial model includes: A visual encoding model that extracts high-level visual features from the input handwritten math formula slices; The image position encoding module and the character position encoding module are used to provide explicit position information for visual features and word embedding vectors respectively; A decoding model, used to output a sequence of character semantic features; A formula tree structure perception module is used to construct the feature vector output by the decoding model into a formula structure tree; The projection layer is used to decode the mathematical formula sequence in an autoregressive manner using the feature vector output by the decoding model.

4. The method according to claim 3, characterized in that The decoding model adopts the Transformer decoding model that introduces the coverage attention mechanism.

5. The method according to claim 3, characterized in that: The processing process of the formula tree structure perception module includes: Input the character semantic feature sequence X extracted by the decoding model; Use the Transformer encoder to extract the character semantic feature sequence X to obtain X ′ ; Through two linear projection functions F c (·) and F p (·), map the feature vector X′ output by the Transformer encoder into child node vectors and parent node vectors respectively; The child node vectors and parent node vectors are combined and added in pairs to obtain the relationship feature matrix M. The vector M at each position in the matrix M is i,j The feature vector representing the relationship between the character with subscript i as the child node and the character with subscript j as the parent node; Use function F score (·) to convert the relationship feature matrix M into the relationship score matrix S, function F score (·) consists of a ReLU activation function and a vector dot product operation; Each element S in the relation score matrix S output by the formula tree structure perception module i,j represents the relationship score with the i-th character as the child node and the j-th character as the parent node, where the higher the score, the greater the possibility of the parent-child relationship. For the element S with the highest score in each row, a,b , it is believed that the model predicts the existence of a parent-child relationship with the ath symbol as the child node and the bth symbol as the parent node, and then constructs the entire mathematical formula tree structure by establishing the relationship between the characters.

6. The method according to claim 1, characterized in that The training strategy of the initial model includes: combining the loss function of the sequence decoding task and the loss function of the tree structure prediction task to train the model; The loss function of the sequence decoding task is: Among them, X t represents the feature vector of the t-th time step output by the decoding model, W o represents the linear projection parameter matrix, b o represents the bias vector; The loss function of the tree structure prediction task is: Among them, S is the relationship score matrix output by the formula tree structure perception module, which is used to estimate the probability that each character t has a parent-child relationship with other characters.

7. The method according to claim 1, characterized in that The method of using the trained handwritten mathematical formula recognition model to perform handwritten mathematical formula recognition includes: Obtain a handwritten mathematical formula image to be recognized; Preprocess the handwritten mathematical formula image to be recognized; The preprocessed handwritten mathematical formula image is input into the trained handwritten mathematical formula recognition model, and the model predicts each symbol in the formula image and generates an initial recognition sequence; A beam search strategy is used for the initial recognition sequence to generate a set of candidate sequences, and a sequence decoding score is calculated for each candidate sequence; The formula tree structure perception module is used to calculate the relationship score matrix of the candidate sequence, and then the tree structure prediction score is calculated for each candidate sequence; The sequence decoding score is added to the tree structure prediction score, and the candidate sequence with the highest comprehensive score is selected as the final output sequence.

8. A handwritten mathematical formula recognition system, characterized in that: include: Annotation data construction module, used to construct annotation data of handwritten mathematical formulas; The model training module is used to construct an initial model for handwritten mathematical formula recognition, use labeled data to train the initial model, and perform joint optimization on the sequence decoding task and the tree structure prediction task to obtain a handwritten mathematical formula recognition model; the formula recognition module is used to use the trained handwritten mathematical formula recognition model to perform handwritten mathematical formula recognition.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a computer, the method according to any one of claims 1 to 7 is implemented.

Images