Handwritten mathematical expression recognition method and system based on self-correction mechanism

Abstract

The invention discloses a handwritten mathematical expression recognition method and system based on a self-correction mechanism. The method comprises the following steps: extracting visual features of an input image; performing preliminary decoding identification on the mathematical expression of the input image to obtain a preliminary decoding sequence; performing text coding on the preliminary decoding sequence to obtain a first text feature, performing calculation according to the similarity between the first text feature and the visual feature to obtain a confidence coefficient vector representing the confidence coefficient of the first text feature, and performing weighted adjustment on the first text feature by using the confidence coefficient vector to obtain a second text feature; and splicing the second text feature with the visual feature to obtain a combined feature, and performing decoding identification on the combined feature again to obtain a mathematical expression identification result. According to the method, the confidence is utilized to guide the method to carry out iterative correction on the preliminary recognition output result, and the formula recognition accuracy can be improved.

Description

Technical Field

[0001] The present invention belongs to the field of pattern recognition, and more specifically, relates to a handwritten mathematical expression recognition method and system based on a self-correction mechanism. Background Art

[0002] Research on handwritten mathematical formula recognition originated from early explorations of character recognition technology. Initially, progress was slow due to technical and theoretical limitations. At the time, methods primarily relied on rule-based and handcrafted features. Researchers meticulously designed various rules to segment and recognize mathematical symbols, such as those based on stroke direction, length, and angle. For structural analysis, this approach relied on pre-defined grammatical rules to logically group mathematical symbols. However, this approach has significant limitations. It is poorly adaptable to diverse handwriting styles, such as differences in stroke thickness and connective strokes due to different writing habits, as well as complex mathematical formula structures. Recognition accuracy is unsatisfactory, and its practical application is subject to numerous restrictions.

[0003] With the emergence of machine learning algorithms, the field of handwritten mathematical formula recognition has ushered in new opportunities. Neural networks have provided a powerful tool for addressing this challenge. The combination of convolutional neural networks (CNNs) and long short-term memory networks (LSTMs) has become the foundation of the encoder-decoder architecture. For example, the WAP model innovatively incorporates an attention mechanism that effectively captures spatial dependencies in handwritten mathematical formulas, allowing the model to focus more on key areas during recognition. DenseWAP, through dense connections, further optimizes feature propagation, alleviates the vanishing gradient problem, and improves model performance.

[0004] Subsequently, the Transformer architecture revolutionized the field of handwritten mathematical formula recognition with its unique self-attention mechanism. Its ability to globally model the contextual dependencies between symbols has led to the emergence of a series of advanced models, including BTTR, CoMER, ICAL, and TAMER. BTTR was the first to apply the Transformer architecture to mathematical formula recognition, breaking the limitations of traditional recurrent neural networks. CoMER introduced the Attention Optimization Module (ARM), addressing the insufficient coverage problem of the attention mechanism and enabling the model to better capture image details. ICAL focuses on inferring implicit characters, improving the recognition of invisible symbols. TAMER integrates a tree perception module, enhancing the ability to predict complex mathematical formula structures.

[0005] Existing technologies for handwritten mathematical formula recognition can be categorized into two types: traditional rule-based and feature engineering approaches, and deep learning-based approaches. However, these technologies still face numerous challenges in coping with complex handwriting styles and understanding the structure of mathematical expressions, particularly when dealing with easily confused characters. The diversity of handwriting styles makes the differences between similar characters more subtle, making it difficult for models to accurately capture these subtle features. Summary of the Invention

[0006] In response to the above-mentioned defects or improvement needs of the prior art, the present invention provides a handwritten mathematical expression recognition method and system based on a self-correction mechanism, the purpose of which is to use confidence to guide itself to iteratively correct the preliminary recognition output results, thereby improving the accuracy of formula recognition.

[0007] To achieve the above object, according to one aspect of the present invention, a method for recognizing handwritten mathematical expressions based on a self-correction mechanism is provided, comprising the steps of:

[0008] Inputting an input image containing a mathematical expression into a visual encoder, wherein the visual encoder is used to extract visual features of the input image;

[0009] Inputting the input image into a decoder, the decoder is used to perform preliminary decoding and recognition on a mathematical expression of the input image to obtain a preliminary decoding sequence;

[0010] Inputting the preliminary decoded sequence and the visual feature into a confidence-guided self-correction module, the self-correction module being configured to perform text encoding on the preliminary decoded sequence to obtain a first text feature, calculating a confidence vector representing the credibility of the first text feature based on similarity between the first text feature and the visual feature, and performing a weighted adjustment on the first text feature using the confidence vector to obtain a second text feature;

[0011] The second text feature is concatenated with the visual feature to obtain a combined feature, and the combined feature is input into the decoder for further decoding and recognition to obtain a mathematical expression recognition result.

[0012] Preferably, the visual encoder adopts DenseNet, and the extracting of visual features of the input image comprises the steps of:

[0013] Performing preliminary visual feature extraction on the input image to obtain preliminary visual features;

[0014] Extracting image position coding features expressing relative position information of symbols in the input image;

[0015] The image position code is added to the preliminary visual feature to serve as the visual feature.

[0016] Preferably, the calculation formula of the visual feature is:

[0017] V feature =DenseNet(I)+E pos

[0018] Among them E pos represents the image position encoding feature, I represents the input image, DenseNet(I) represents the preliminary visual feature, V feature Represents the visual feature.

[0019] Preferably, the text encoding of the preliminary decoding sequence to obtain the first text feature comprises the steps of:

[0020] The preliminary decoded sequence is subjected to text embedding operation to obtain a high-dimensional vector, denoted as T emb ;

[0021] Extract the text position code of the preliminary decoding sequence, denoted as T pos ;

[0022] In the high-dimensional vector T emb Add text position code T pos Then input it into the Transformer encoder to obtain the first text feature.

[0023] Preferably, the step of calculating the confidence vector representing the credibility of the first text feature based on the similarity between the first text feature and the visual feature comprises the following steps:

[0024] The first text feature is recorded as T feature , the visual feature is recorded as V feature , T feature and V feature Linear projection into the same space gives T feature The projection vector T p and V feature The projection vector V p ;

[0025] Calculate the projection vector T in sequence p Each text word and projection vector V p The similarity of each visual area in the text is calculated to obtain the similarity score of each text word;

[0026] The similarity score of each text word is input into the feedforward network to obtain the confidence score of each text word. The confidence scores of all text words constitute a confidence vector representing the credibility of the first text feature.

[0027] Preferably, the projection vector T p Each text word and projection vector V p The calculation formula for the similarity of each visual area in is:

[0028]

[0029] Among them, S i,j represents the similarity between the i-th text word and the j-th visual area, and L represents the projection vector T p The number of Chinese text words, H×W represents the projection vector V p The number of visual areas in represents the i-th text word, represents the jth visual area, d model express Dimensions;

[0030] For the i-th text word in the text sequence, the maximum value is selected from the similarities between the i-th text word and all visual areas as the similarity score of the i-th text word:

[0031]

[0032] Among them, S max,i Represents the similarity score of the i-th text word.

[0033] Preferably, the calculation formula for weighted adjustment of the first text feature using the confidence vector is:

[0034] T′ feature =T feature ⊙CS

[0035] Among them, T′ feature represents the second text feature, T feature represents the first text feature, CS represents the confidence vector, and ⊙ represents a dot product operation.

[0036] Preferably, the visual encoder, the self-correction module, and the decoder are jointly trained, and the total loss function calculation formula of the joint training is:

[0037] L total =L initial +L refined +L cs

[0038] Among them, L total Denotes the total loss function of joint training, L initial represents the initial prediction loss identified by the decoder, L refinedIndicates the corrected prediction loss of the decoder for decoding recognition again, L initual , L refined The standard cross entropy loss function is used for calculation, L cs represents the confidence score loss;

[0039] Confidence score loss L cs A focal loss is used to supervise the prediction confidence score of each tag predicted by the self-correction module. Where 0 indicates the lowest confidence score, that is, the probability that the tag is wrong is the highest; 1 indicates the highest confidence score, that is, the probability that the tag is correct is the highest. The binary true label p for each tag is generated by aligning the initial prediction with the true label. i ∈{0,1}, where 0 indicates a wrong prediction and 1 indicates a correct prediction. The calculation formula is:

[0040]

[0041] in represents the prediction confidence score of the i-th token, α represents the weighting factor, γ represents the focusing parameter, and L represents the number of binary true labels.

[0042] Preferably, during the joint training process, if the token with the highest predicted probability and the token with the second highest predicted probability for the t-th text word are less than a preset threshold τ, the model will automatically replace the token with the second highest predicted probability. The calculation formula is:

[0043]

[0044] in, represents the tag with the highest predicted probability for the t-th text word, represents the second highest predicted probability tag of the t-th text word, and y represents the output tag.

[0045] According to another aspect of the present invention, a handwritten mathematical expression recognition system based on a self-correction mechanism is provided, comprising:

[0046] a visual encoder configured to extract visual features of an input image comprising a handwritten mathematical expression;

[0047] A decoder, configured to perform preliminary decoding and recognition on the mathematical expression of the input image to obtain a preliminary decoding sequence;

[0048] A confidence-guided self-correction module is configured to perform text encoding on the preliminary decoded sequence to obtain a first text feature, calculate a confidence vector representing the credibility of the first text feature based on the similarity between the first text feature and the visual feature, and perform a weighted adjustment on the first text feature using the confidence vector to obtain a second text feature;

[0049] The decoder is further configured to concatenate the second text feature with the visual feature to obtain a combined feature, and input the combined feature into the decoder for further decoding and recognition to obtain a mathematical expression recognition result.

[0050] In general, compared with the prior art, the above technical solution conceived by the present invention can analyze the front wheel output of the model to detect possible errors, and then correct the results again to obtain a more reliable output. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 Schematic diagram of the working principle of the handwritten mathematical expression recognition method according to an embodiment of the present invention;

[0052] Figure 2 is a schematic diagram of a confidence scoring module according to an embodiment of the present invention;

[0053] Figure 3 Schematic diagram of iterative self-correction of a model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0054] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only intended to illustrate the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0055] In the description of the embodiments of this application, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly specifying the number of the technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of such features. "Multiple" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0056] The terms "including" and "having" and any variations thereof in the embodiments of the present invention are intended to cover non-exclusive inclusions. For example, a process, method, apparatus, product or device comprising a series of steps or modules is not necessarily limited to those steps or modules explicitly listed, but may include other steps or modules not explicitly listed or inherent to these processes, methods, products or devices.

[0057] The naming or numbering of the steps in the embodiments of the present invention does not mean that the steps in the method flow must be executed in the time / logical sequence indicated by the naming or numbering. The execution order of the named or numbered process steps can be changed according to the technical purpose to be achieved, as long as the same or similar technical effects can be achieved.

[0058] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present invention. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute a separate or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0059] Existing handwritten formula recognition technology lacks the ability to self-correct. Iterative self-correction is a hallmark of human complex problem-solving. Humans can improve by focusing on potential errors in their initial interpretation. Similarly, intelligent recognition systems can achieve more reliable outputs by analyzing the model's previous outputs to detect potential errors and then re-correcting the results.

[0060] The present invention provides a method and system for handwritten mathematical expression recognition based on a self-correction mechanism to solve the problem that handwritten mathematical expression recognition models lack self-correction capabilities.

[0061] A method for recognizing handwritten mathematical expressions based on a self-correction mechanism according to an embodiment of the present invention works as follows: Figure 1 As shown, it includes steps S1 to S4.

[0062] S1, inputs the input image containing the mathematical expression into the visual encoder, which is used to extract the visual features of the input image.

[0063] Preferably, DenseNet is used as a visual encoder to extract features from the input image. DenseNet strengthens the information transfer between different layers in the network through dense connections, thereby extracting rich visual features. Then, position encoding features are further added on this basis, so that the model can perceive the spatial position relationship of features in the image. This is very important for understanding the spatial layout of symbols in mathematical expressions. Position encoding is constructed by combining sine and cosine functions. The visual features extracted by DenseNet are the same size as the position encoding features. The two feature matrices are added together to obtain the output of the visual encoder.

[0064] V feature =DenseNet(I)+E pos

[0065] Among them E pos Represents the position coding feature, which is used to represent the relative position information of the symbols in the input image. I represents the input mathematical formula image. Represents the extracted visual feature map. This feature map can be divided into H×W visual regions, each of which corresponds to a local receptive field in the image and is used for subsequent similarity calculations with text features.

[0066] S2, inputting the input image into the decoder, which is used to perform preliminary decoding and recognition on the mathematical expression of the input image to obtain a preliminary decoding sequence.

[0067] Due to the structural complexity of handwritten mathematical expressions and the presence of similar characters, the initial decoded sequence obtained by this preliminary decoding recognition may contain errors. Further corrections will be made based on this in subsequent steps.

[0068] The preliminary decoding and recognition refers to the decoding and recognition performed again in step S4, and does not necessarily refer to the first decoding and recognition. There can be multiple rounds of decoding outputs, and the recognition results of the previous round are continuously iterated and modified, and the previous round of recognition is used as the preliminary decoding and recognition.

[0069] S3, the preliminary decoded sequence and visual features V feature Input to the confidence-guided self-correction module (CGSCM), which is used to encode the preliminary decoding sequence to obtain the first text feature T feature , according to the first text feature T feature and visual features V feature The confidence vector representing the credibility of the first text feature is obtained by similarity calculation, and the confidence vector is used to calculate the credibility of the first text feature T feature Perform weighted adjustment to obtain the second text feature T′ feature .

[0070] Preferably, the confidence-guided self-correction module consists of a Transformer for text encoding and a confidence scoring component, and the network structure of the confidence scoring component is as follows: Figure 2 This module obtains a confidence vector based on the similarity comparison between the previous round of decoding output of the model and the visual features.

[0071] In the first step, the Transformer encoder is used to encode the text of the previous round of decoding output. Given the previous round of decoding output sequence T∈R L×1 (L is the sequence length), first the text embedding operation is performed to obtain a high-dimensional vector Then, add the text position code T pos Finally, the Transformer encoder captures local and long-range dependencies within the sequence, which is represented as follows:

[0072] T feature = TransformerEncoder(T emb +T pos )

[0073] Among them, T feature ∈R L×dmodel Represents the first text feature extracted.

[0074] The second step is based on the first text feature T obtained above. feature and visual features V feature , calculate the confidence score. First, T feature and V feature Linear projection into the same space gives T feature The projection vector and V feature The projection vector In order to measure the similarity. Then, Figure 2 As shown in Figure 2, the similarity measurement method is used to calculate the similarity between the i-th text token and the j-th visual area:

[0075]

[0076] in and d represents the i-th text tag and the j-th visual area respectively model dimensional feature vector, L represents the projection vector T p The number of Chinese text words, H×W represents the projection vector V p The number of visual regions in the similarity matrix S∈R H×W×L It reflects the similarity between each textual token and each visual region.

[0077] For the i-th token in the text sequence, the maximum value is selected from the similarities between the i-th text token and all visual regions as the similarity score of the i-th text token:

[0078]

[0079] A correctly recognized token will show a significantly high similarity with a specific visual region, while a misrecognized token will have a relatively uniform similarity metric score with multiple regions. Finally, the similarity score is input into a feedforward network to generate a confidence score for the i-th token. The generator is defined as follows:

[0080] CS i =σ(W2·ReLU(W1·S max,i +b1)+b2)

[0081] Where W2 and W1 represent weight matrices, b1 and b2 are bias terms, σ(·) represents the sigmoid activation function, and CS i Represents the confidence score of the i-th token. A lower score indicates a higher uncertainty about the correctness of the token.

[0082] The confidence scores of all tokens form the confidence vector CS.

[0083] The third step is to use the confidence vector CS to guide the model to focus on characters with high uncertainty. Specifically, the confidence vector CS∈R L×1 With the first text feature T feature Perform element-by-element multiplication to obtain the reweighted second text feature T′ feature :

[0084] T′ feature =T feature ⊙CS

[0085] Where ⊙ represents the dot multiplication operation (d is the confidence value of each position copied during the dot multiplication process). model times, to multiply element-wise with the corresponding eigenvector).

[0086] S4, the second text feature T′ feature With visual features V feature The combined features are concatenated and input into the decoder for further decoding and recognition to obtain the mathematical expression recognition results.

[0087] Preferably, the output T′ of the CGSCM module feature With Vfeature Concatenate (concat) to form a combined feature tensor, which is input into the Transformer decoder for further decoding:

[0088]

[0089] Y^=TransformerDecoder(X^)

[0090] Preferably, the decoder introduces an Attention Redistribution Module (ARM) to enhance the attention mechanism and generates the corrected LaTeX sequence in an autoregressive manner, generating output tokens one by one.

[0091] Preferably, if Figure 3 As shown in Figure 5, the visual encoder, self-correction module, and decoder are jointly trained as one model.

[0092] Training strategy. The training of the model is divided into an initial stage and an optimization stage. In the initial stage, the model generates a preliminary LaTeX sequence prediction based on the input image features. Due to the structural complexity of handwritten mathematical expressions and the presence of similar characters, this initial prediction may contain errors. In the optimization stage, the model uses CGSCM to compare and analyze the initial predictions. CGSCM calculates the confidence score of each token and identifies low-confidence tokens. Then, the corresponding text features are reweighted according to these confidence scores, and the reweighted text features are connected with the original visual features and input into the Transformer Decoder to generate refined predictions. This iterative training method allows the model to continuously learn from its own prediction results and gradually improve its ability to recognize and correct errors.

[0093] As training progresses, the probability of the model generating correct predictions will gradually increase, which will lead to the problem of a reduction in the number of error samples encountered by the model in the later stages of training, which is not conducive to learning how to identify and correct errors. In order to solve this problem, the present invention preferably proposes a simple replacement strategy. During the training process, if the difference between the marker with the highest predicted probability (TOP 1) and the marker with the second highest predicted probability (TOP 2) is less than a preset threshold τ, the model will automatically replace the marker with the highest predicted probability (TOP 1) with the marker with the second highest predicted probability (TOP 2).

[0094]

[0095] in, and y represents the token with the highest predicted probability (TOP 1) and the token with the second highest predicted probability (TOP 2) for the tth token, and y represents the output token. This intervention strategy aims to force the model to distinguish easily confused symbols, allowing the model to continuously strengthen its ability to distinguish similar symbols during training and further improve its self-optimization capabilities.

[0096] Preferably, the training process adopts an end-to-end joint optimization strategy, setting three loss components to optimize the text generation and error detection tasks simultaneously. The total loss function is as follows:

[0097] L total =L initial +L refined +L cs

[0098] Initial prediction loss L initial and the corrected prediction loss L refined The standard cross entropy loss function is used to calculate the difference between the true value and the predicted value. The cross entropy loss function is defined as follows:

[0099]

[0100] Among them, y c represents the c-th class label, p c Represents the c-th class prediction value, where C is the total number of classes. Confidence score loss L cs A focal loss is used to supervise the confidence score of each tag predicted by the Confidence Scoring Module (CSM) Where 0 represents the lowest confidence score, that is, the probability that the tag is wrong is the highest; 1 represents the highest confidence score, that is, the probability that the tag is correct is the highest. The binary true label p for each tag is generated by aligning the initial prediction with the true label i ∈{0,1}, where 0 indicates a wrong prediction and 1 indicates a correct prediction. This loss emphasizes hard examples by reweighting the incorrectly predicted labels:

[0101]

[0102] in is the predicted confidence score of the i-th token, α is a weighting factor used to adjust the importance of positive and negative examples, γ is a focusing parameter used to control the modulation strength of simple samples, and L represents the number of binary true labels, that is, the number of tokens in the previous text.

[0103] A handwritten mathematical expression recognition system based on a self-correction mechanism according to an embodiment of the present invention includes:

[0104] a visual encoder for extracting visual features of an input image from an input image containing a handwritten mathematical expression;

[0105] A decoder is used to perform preliminary decoding and recognition on the mathematical expression of the input image to obtain a preliminary decoding sequence;

[0106] A confidence-guided self-correction module is configured to perform text encoding on the preliminary decoding sequence to obtain a first text feature, calculate a confidence vector representing the credibility of the first text feature based on the similarity between the first text feature and the visual feature, and perform a weighted adjustment on the first text feature using the confidence vector to obtain a second text feature;

[0107] The decoder is further used to concatenate the second text feature with the visual feature to obtain a combined feature, and input the combined feature into the decoder for further decoding and recognition to obtain a mathematical expression recognition result.

[0108] The working principle and technical effects of the handwritten mathematical expression recognition system are the same as those of the above-mentioned handwritten mathematical expression recognition method, and will not be repeated here.

[0109] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A handwritten mathematical expression recognition method based on a self-correction mechanism, characterized in that: Including steps: Inputting an input image containing a mathematical expression into a visual encoder, wherein the visual encoder is used to extract visual features of the input image; Inputting the input image into a decoder, the decoder is used to perform preliminary decoding and recognition on a mathematical expression of the input image to obtain a preliminary decoding sequence; Inputting the preliminary decoded sequence and the visual feature into a confidence-guided self-correction module, the self-correction module being configured to perform text encoding on the preliminary decoded sequence to obtain a first text feature, calculating a confidence vector representing the credibility of the first text feature based on similarity between the first text feature and the visual feature, and performing a weighted adjustment on the first text feature using the confidence vector to obtain a second text feature; The second text feature is concatenated with the visual feature to obtain a combined feature, and the combined feature is input into the decoder for further decoding and recognition to obtain a mathematical expression recognition result.

2. A handwritten mathematical expression recognition method based on a self-correction mechanism as claimed in claim 1, characterized in that: The visual encoder adopts DenseNet, and the extraction of visual features of the input image includes the steps of: Performing preliminary visual feature extraction on the input image to obtain preliminary visual features; Extracting image position coding features expressing relative position information of symbols in the input image; The image position code is added to the preliminary visual feature to serve as the visual feature.

3. A handwritten mathematical expression recognition method based on a self-correction mechanism as claimed in claim 2, characterized in that: The calculation formula of the visual feature is: V feature =DenseNet(I)+E pos Among them E pos represents the image position encoding feature, I represents the input image, DenseNet(I) represents the preliminary visual feature, V feature Represents the visual feature.

4. A handwritten mathematical expression recognition method based on a self-correction mechanism as claimed in claim 1, characterized in that: The text encoding of the preliminary decoding sequence to obtain the first text feature comprises the steps of: The preliminary decoded sequence is subjected to text embedding operation to obtain a high-dimensional vector, denoted as T emn ; Extract the text position code of the preliminary decoding sequence, denoted as T pos ; In the high-dimensional vector T emb Add text position code T pos Then input it into the Transformer encoder to obtain the first text feature.

5. A handwritten mathematical expression recognition method based on a self-correction mechanism as claimed in claim 1, characterized in that: The step of obtaining a confidence vector representing the credibility of the first text feature by calculating the similarity between the first text feature and the visual feature comprises the following steps: The first text feature is recorded as T feature , the visual feature is recorded as V feature , T featufe and V feature Linear projection into the same space gives T feature The projection vector T p and V feature The projection vector V p ; Calculate the projection vector T in sequence p Each text word and projection vector V p The similarity of each visual area in the text is calculated to obtain the similarity score of each text word; The similarity score of each text word is input into the feedforward network to obtain the confidence score of each text word. The confidence scores of all text words constitute a confidence vector representing the credibility of the first text feature.

6. A handwritten mathematical expression recognition method based on a self-correction mechanism as claimed in claim 5, characterized in that: The projection vector T p Each text word and projection vector V p The calculation formula for the similarity of each visual area in is: Among them, S i,j represents the similarity between the i-th text word and the j-th visual area, and L represents the projection vector T p The number of Chinese text words, H×W represents the projection vector V p The number of visual areas in represents the i-th text word, represents the jth visual area, d model express Dimensions; For the i-th text word in the text sequence, the maximum value is selected from the similarities between the i-th text word and all visual areas as the similarity score of the i-th text word: Among them, S max,i Represents the similarity score of the i-th text word.

7. A handwritten mathematical expression recognition method based on a self-correction mechanism as claimed in claim 1, characterized in that: The calculation formula for weighted adjustment of the first text feature using the confidence vector is: T′ feature =T feature ⊙CS Among them, T′ feature represents the second text feature, T feature represents the first text feature, CS represents the confidence vector, and ⊙ represents a dot product operation.

8. A handwritten mathematical expression recognition method based on a self-correction mechanism as claimed in claim 1, characterized in that: The visual encoder, the self-correction module, and the decoder are jointly trained. The total loss function of the joint training is calculated as follows: L total =L initial +L refined +L cs Among them, L total Denotes the total loss function of joint training, L initial represents the initial prediction loss identified by the decoder, L refined Indicates the corrected prediction loss of the decoder for decoding recognition again, L initial , L refined The standard cross entropy loss function is used for calculation, L cs represents the confidence score loss; Confidence score loss L cs A focal loss is used to supervise the prediction confidence score of each tag predicted by the self-correction module. Where 0 indicates the lowest confidence score, that is, the probability that the tag is wrong is the highest; 1 indicates the highest confidence score, that is, the probability that the tag is correct is the highest. The binary true label p for each tag is generated by aligning the initial prediction with the true label. i ∈{0,1}, where 0 indicates a wrong prediction and 1 indicates a correct prediction. The calculation formula is: in represents the prediction confidence score of the i-th token, α represents the weighting factor, γ represents the focusing parameter, and L represents the number of binary true labels.

9. A handwritten mathematical expression recognition method based on a self-correction mechanism as claimed in claim 8, characterized in that: During the joint training process, if the difference between the token with the highest predicted probability and the token with the second highest predicted probability for the t-th text word is less than the preset threshold τ, the model will automatically replace the token with the second highest predicted probability. The calculation formula is: in, represents the tag with the highest predicted probability for the t-th text word, represents the second highest predicted probability tag of the t-th text word, and y represents the output tag.

10. A handwritten mathematical expression recognition system based on a self-correction mechanism, characterized in that: include: a visual encoder configured to extract visual features of an input image comprising a handwritten mathematical expression; A decoder, configured to perform preliminary decoding and recognition on the mathematical expression of the input image to obtain a preliminary decoding sequence; A confidence-guided self-correction module is configured to perform text encoding on the preliminary decoded sequence to obtain a first text feature, calculate a confidence vector representing the credibility of the first text feature based on the similarity between the first text feature and the visual feature, and perform a weighted adjustment on the first text feature using the confidence vector to obtain a second text feature; The decoder is further configured to concatenate the second text feature with the visual feature to obtain a combined feature, and input the combined feature into the decoder for further decoding and recognition to obtain a mathematical expression recognition result.

Images