Dynamic recognition method of handwritten mathematical formulas based on multimodal feature fusion

Abstract

The present invention discloses a method for dynamic recognition of handwritten mathematical formulas based on multimodal feature fusion, comprising the following steps: 1. encoding and converting the stroke trajectory sequence information contained in the data using the HSV color model to generate an HSV pseudo-color image, and then fusing the HSV pseudo-color image with the data binary image to obtain a temporal rich image; 2. extracting features from the temporal rich image and the stroke trajectory sequence using a convolutional neural network to obtain image features and stroke features, and enhancing the stroke features using the image features, and then fusing the image features with the enhanced stroke features to obtain multimodal features; 3. enhancing the input multimodal features using position encoding using a decoder, and then performing symbol prediction in combination with an attention mechanism and a multi-scale symbol counting module to generate a mathematical formula symbol sequence; and 4. designing rules to correct the generated mathematical formula symbol sequence. The present invention can improve the accuracy of easily confused character recognition.

Description

Technical Field

[0001] The present invention relates to the technical field of handwritten mathematical formula recognition, and in particular to a handwritten mathematical formula dynamic recognition method based on multimodal feature fusion. Background Art

[0002] In the field of handwritten mathematical formula recognition, traditional algorithms often face the problem of low accuracy in recognizing easily confused symbols. This is particularly true when dealing with morphologically and visually similar symbols, such as the plus and multiplication signs, or letters and numbers. Existing technologies often fail to accurately distinguish these symbols. Existing solutions primarily rely on single-modal information, such as images or strokes. However, these methods often fail to fully utilize stroke order or contextual information when dealing with the ambiguity of handwritten symbols, leading to misrecognition. Summary of the Invention

[0003] To address the above-mentioned technical problems in the prior art, the present invention proposes a dynamic recognition method for handwritten mathematical formulas based on multimodal feature fusion. This method uses a heterogeneous dual-branch spatiotemporal fusion network for easily confused symbol recognition. This network combines image features with stroke features, performing deep fusion in both spatial and temporal dimensions, thereby improving the recognition accuracy of easily confused symbols. The specific technical solution is as follows:

[0004] A dynamic recognition method for handwritten mathematical formulas based on multimodal feature fusion, comprising:

[0005] Step 1: Obtain handwritten mathematical formula data, use the HSV color model to encode the stroke trajectory sequence information contained in the handwritten mathematical formula data, generate an HSV pseudo-color image, then fuse the HSV pseudo-color image with the binary image contained in the handwritten mathematical formula data to obtain a time-series enriched image, and then input the time-series enriched image into a convolutional neural network;

[0006] Step 2: Using a convolutional neural network, feature extraction is performed on the time-rich image and the stroke trajectory sequence contained in the mathematical formula data to obtain image features and stroke features. The stroke features are enhanced using the image features, and then the image features and the enhanced stroke features are fused to obtain multimodal features.

[0007] Step 3: Through the decoder, position encoding is used to enhance the multimodal features of the input, and then the attention mechanism and multi-scale symbol counting module are combined to perform symbol prediction to generate a mathematical formula symbol sequence;

[0008] Step 4: Design rules to modify the generated mathematical formula symbol sequence.

[0009] Furthermore, in step 1, the stroke trajectory sequence information contained in the handwritten mathematical formula data is encoded and converted using the HSV color model, specifically: the writing order of each stroke, the stroke length and the complexity of the formula are respectively mapped to the hue, saturation and value of the HSV model to generate a pseudo-color image, that is, the writing order of the strokes is mapped to the hue value, the stroke length is associated with the saturation, and the number of strokes is normalized and combined with the value to generate an image with time series information.

[0010] Furthermore, in step 2, a DenseNet-BC network is used to extract features from the temporally rich image to obtain image features, and a selective state space model mamba is used to process the stroke trajectory sequence to obtain stroke features.

[0011] Furthermore, in step 2, the image features and stroke features are combined, and a priori alignment relationship is established through the coordinate sequence of the image and the strokes. Then, the expressive power of the stroke features is enhanced through feature conversion and resetting operations.

[0012] Furthermore, in step 2, the image features are fused with the enhanced stroke features. First, a one-dimensional convolutional layer is used to align the image and stroke features to ensure that the feature sizes of the two modalities are consistent. Then, a Concat operation is used to concatenate the image features and stroke features in the channel dimension, thereby increasing the number of features.

[0013] Furthermore, in step 3, the decoder performs dimensionality transformation on the input multimodal features through a 1×1 convolutional layer, and then adds position encoding for feature enhancement.

[0014] Furthermore, the multi-scale symbol counting module performs multi-scale symbol counting processing on the image features to predict various symbol count values, that is, statistics for various symbols from a global perspective.

[0015] Furthermore, the attention mechanism selects and weights feature maps of different scales, and generates a context vector by fusing low-resolution and high-resolution features.

[0016] Furthermore, the decoder calculates the probability distribution of the current symbol based on the current hidden layer state, the context vector, the count values ​​of various symbols and the prediction result of the previous symbol, selects the symbol with the highest probability through the probability distribution, and outputs it, gradually generating a complete mathematical formula symbol sequence. The previous hidden layer state uses the previous symbol, takes its feature code and inputs it into the decoder GRU layer to obtain it.

[0017] In step 4, according to the probability distribution of symbol recognition, combined with the statistical results and confusion matrix, the generated mathematical formula symbol sequence is corrected through LaTeX grammar rules.

[0018] Beneficial effects: The method of the present invention improves the accuracy of recognizing easily confused characters by combining image features and stroke features, uses contextual information to enhance symbol category guidance, uses temporal information to enrich image representation for initial alignment, and introduces stroke feature enhancement for detail distinction. It can effectively distinguish and re-identify similar symbols written by users, thereby improving the recognition accuracy; and, based on the statistical probability of recognition test results, a similar symbol correction method is proposed, and output rules are set to distinguish them to enhance the model's ability to distinguish such symbols, thereby improving user writing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 1 is a network architecture diagram of a method for dynamic recognition of handwritten mathematical formulas based on multimodal feature fusion according to an embodiment of the present invention;

[0020] Figure 2 This is a time-series enriched image effect diagram of an embodiment of the present invention;

[0021] Figure 3 2. This is a schematic diagram of a network structure for enhancing stroke features according to an embodiment of the present invention;

[0022] Figure 4 is a schematic diagram of a decoder incorporating a multi-scale symbol counting module according to an embodiment of the present invention;

[0023] Figure 5 2 is a schematic diagram of a multi-scale symbol counting module according to an embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and technical effect of the present invention more clear, the present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0025] like Figure 1 As shown, this embodiment discloses a method for dynamic recognition of handwritten mathematical formulas based on multimodal feature fusion, including:

[0026] Step 1: Figure 2 As shown in the figure, handwritten mathematical formula data is obtained, and the stroke trajectory sequence information in the data is encoded and converted using the HSV color model to generate an HSV pseudo-color image with time sequence information of the formula strokes. The HSV pseudo-color image is fused with the binary image in the data to obtain a time sequence enriched image, which is then input into a convolutional neural network for feature processing.

[0027] Specifically, each handwritten mathematical formula data includes an image and corresponding stroke trajectory sequence information. The image of the handwritten formula is generally a binary image, and the stroke sequence consists of point coordinates and stroke pause marks.

[0028] The HSV color model is used to encode and convert the stroke trajectory sequence information in the data, specifically: the writing order, stroke length and formula complexity of each stroke are mapped to the hue, saturation and value of the HSV model respectively, to generate a pseudo-color image, that is, the writing order of the strokes is mapped to the hue value, the stroke length is associated with the saturation, the number of strokes is normalized and combined with the value to generate an image with time series information.

[0029] The generated HSV pseudo-color image is fused with the original binary image as the input image to ensure that the convolutional neural network can simultaneously process the spatial features of the image and the temporal features of the strokes.

[0030] Step 2: Figure 3 As shown in the figure, through the convolutional neural network, the time-rich image and the stroke trajectory sequence are respectively extracted to obtain image features and stroke features, and the image features are used to enhance the stroke features, and then the image features are fused with the enhanced stroke features to obtain multimodal features.

[0031] Specifically, a DenseNet-BC network is used to extract features from the temporally rich image to obtain image features. The DenseNet-BC network, through its dense connections and bottleneck layer design, can retain low-level detail information while also capturing deep semantic information when extracting image features.

[0032] The selective state-space model (Mamba) is used to process stroke trajectory sequences and obtain stroke features. The selective state-space model captures the long-term temporal dependencies of strokes through dynamic modeling of hidden states, making it suitable for processing both local dynamics and global correlations in stroke writing sequences.

[0033] In the initial alignment stage of image and stroke sequences, a priori alignment relationship is established through the coordinate sequences of images and strokes to help the network capture cross-modal dependencies.

[0034] The feature maps of time-rich images extracted by convolutional neural networks are combined with the stroke features extracted by selective state-space models. Through feature conversion and reset operations, the expressive power of stroke features is enhanced, ensuring that the network can better capture detailed information when processing handwritten formulas.

[0035] The described fusion of image features and enhanced stroke features first uses a one-dimensional convolutional layer to align the image and stroke features to ensure that the feature sizes of the two modalities are consistent, and then uses a Concat operation to splice the image features and stroke features in the channel dimension, thereby increasing the number of features and allowing the model to learn more feature representations of the same formula.

[0036] By introducing temporal enrichment into images, the team leveraged the order of strokes and the contextual relationships between them to enrich the images, helping convolutional neural networks better process dynamic information within the images. By combining the spatial information of the image with the temporal information of the strokes, they established a priori alignment relationships, helping the network capture cross-modal dependencies and significantly improving the ability to recognize easily confused symbols.

[0037] Step 3: Figure 4 As shown, the multimodal features are input into the decoder, enhanced by position encoding, and then combined with the attention mechanism and multi-scale symbol counting module for symbol prediction to generate a mathematical formula symbol sequence.

[0038] Specifically, the present invention uses a decoder that integrates a multi-scale symbol counting module to map the feature information of the image and strokes into the final handwritten mathematical formula symbols. The specific process includes:

[0039] Input feature processing: The input multimodal features are transformed into intermediate layer features through a 1×1 convolution layer to ensure that the number of channels of each feature map is consistent.

[0040] Positional encoding enhancement: To better capture the spatial relationship between symbols, the decoder adds positional encoding to the input features, enhancing the input feature map and providing spatial position information for distinguishing symbols at different spatial locations. This enhances the decoder's ability to distinguish between different positions in the image. This is particularly important in handwritten mathematical formulas. For example, in fractions or square roots, the hierarchical relationship between symbols is very important, and positional encoding can help the decoder identify these relationships.

[0041] Symbol counting perception: e.g. Figure 5 As shown, multi-scale symbol counting processing is performed on the image features to predict various symbol count values, that is, statistics for various symbols.

[0042] To address the problem of symbol recognition in complex formulas, the decoder of this invention incorporates a multi-scale symbol counting module. This module combines local symbol features with global symbol category statistics to provide more accurate contextual information for each symbol prediction, helping to identify complex formulas and effectively reducing the likelihood of duplicate or missed symbols. This module determines the meaning and location of symbols based on contextual information at different scales, making the decoding process more responsive to complex symbols and long formulas.

[0043] Attention Weighting: The attention mechanism is used to select and weight feature maps of different scales, and generate a refined context vector by fusing low-resolution and high-resolution features for accurately predicting the next symbol.

[0044] During decoding, attention weights are calculated based on the context of the current symbol, and different focus points are assigned based on the weights, ensuring that the decoder focuses on the areas that are most important for predicting the current symbol. This mechanism enables the decoder to focus on local features that are critical for symbol recognition.

[0045] Symbol sequence decoding output: At each time step, the generated context vector is input into a series of GRU layers. These layers calculate the probability distribution of the current symbol based on the current hidden layer state, the context vector, the counts of various symbols, and the prediction of the previous symbol. The symbol with the highest probability is selected from this probability distribution and output, gradually generating a complete mathematical formula symbol sequence. The previous hidden layer state is obtained by taking the feature code of the previous symbol and inputting it into the GRU layer.

[0046] The decoder of this embodiment not only focuses on local features, but also further adjusts the prediction result of each symbol through the count vector of the global context, thereby reducing symbol duplication and misrecognition.

[0047] Step 4: Modify the generated mathematical formula symbol sequence based on LaTeX syntax rules and the probability distribution of symbols, as follows.

[0048] Rule-based correction method: After symbol recognition, a rule-based correction method is designed to correct common and easily confused symbols, further improving the model's performance in complex formulas. For example, for misidentification between "0" and "O", the following rules are set:

[0049] If the string contains both letters and ◯ symbols, replace all occurrences of ◯ with the letter "O". If the string contains only ◯ symbols, replace them with the number "0".

[0050] Probability distribution correction method:

[0051] According to the probability distribution of symbol recognition, combined with statistical results and confusion matrix, the recognition results are corrected through LaTeX grammar rules to improve the discrimination of easily confused symbols.

[0052] LaTeX syntax rule corrections:

[0053] According to LaTeX syntax rules, the recognition results are post-processed to ensure the correct output of symbols. For example, to deal with the misrecognition problem of brackets and the number "1", the brackets are checked to see if they are closed. If the left bracket is not closed, it is replaced with "1".

[0054] Symbol output: After the corrected symbolic results are output, the final result is a mathematical formula text that conforms to the LaTeX syntax rules. At this point, the recognition result contains all the correct symbols and meets the syntax requirements, making it suitable for further formula parsing and calculation.

[0055] Result verification and display:

[0056] Verify the recognition results, especially evaluate the accuracy of distinguishing easily confused symbols, to ensure that the algorithm has high recognition accuracy in different scenarios.

[0057] The above description is only a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Although the implementation process of the present invention is described in detail above, it is still possible for those familiar with the art to modify the technical solutions described in the above examples or to replace some of the technical features therein with equivalents. Any modifications, equivalent replacements, etc. made within the spirit and principles of the present invention shall be included in the scope of protection of the present invention.

Claims

1. A dynamic recognition method for handwritten mathematical formulas based on multimodal feature fusion, characterized in that: include: Step 1: Obtain handwritten mathematical formula data, use the HSV color model to encode the stroke trajectory sequence information contained in the handwritten mathematical formula data, generate an HSV pseudo-color image, then fuse the HSV pseudo-color image with the binary image contained in the handwritten mathematical formula data to obtain a time-series enriched image, and then input the time-series enriched image into a convolutional neural network; Step 2: Using a convolutional neural network, we extract features from the temporally rich image and the stroke trajectory sequence to obtain image features and stroke features. We then use the image features to enhance the stroke features, and then fuse the image features with the enhanced stroke features to obtain multimodal features. Step 3: Through the decoder, position encoding is used to enhance the multimodal features of the input, and then the attention mechanism and multi-scale symbol counting module are combined to perform symbol prediction to generate a mathematical formula symbol sequence; Step 4: Design rules to modify the generated mathematical formula symbol sequence; In step 1, the HSV color model is used to encode the stroke trajectory sequence information contained in the handwritten mathematical formula data, specifically: The writing order, length and formula complexity of each stroke are mapped to the hue, saturation and value of the HSV model respectively to generate a pseudo-color image. That is, the writing order of the strokes is mapped to the hue value, the stroke length is associated with the saturation, and the number of strokes is normalized and combined with the value to generate an image with time series information.

2. The method for dynamic recognition of handwritten mathematical formulas according to claim 1, wherein: In step 2, the DenseNet-BC network is used to extract features from the temporal rich image to obtain image features, and the selective state space model mamba is used to process the stroke trajectory sequence to obtain stroke features.

3. The method for dynamic recognition of handwritten mathematical formulas according to claim 1, wherein: In step 2, the image features and stroke features are combined, and a priori alignment relationship is established through the coordinate sequence of the image and the strokes. Then, the expressiveness of the stroke features is enhanced through feature conversion and resetting operations.

4. The method for dynamic recognition of handwritten mathematical formulas according to claim 1, wherein: In step 2, the image features are fused with the enhanced stroke features. First, a one-dimensional convolutional layer is used to align the image and stroke features to ensure that the feature sizes of the two modalities are consistent. Then, a Concat operation is used to concatenate the image features and stroke features in the channel dimension, thereby increasing the number of features.

5. The method for dynamic recognition of handwritten mathematical formulas according to claim 1, wherein: In step 3, the decoder transforms the dimension of the input multimodal features through a 1×1 convolutional layer, and then adds position encoding for feature enhancement.

6. The method for dynamic recognition of handwritten mathematical formulas according to claim 5, wherein: The multi-scale symbol counting module performs multi-scale symbol counting processing on the image features to predict various symbol count values, that is, statistics for various symbols from a global perspective.

7. The method for dynamic recognition of handwritten mathematical formulas according to claim 6, wherein: The attention mechanism selects and weights feature maps of different scales and generates a context vector by fusing low-resolution and high-resolution features.

8. The method for dynamic recognition of handwritten mathematical formulas according to claim 7, wherein: The decoder calculates the probability distribution of the current symbol based on the current hidden layer state, context vector, various symbol count values ​​and the prediction result of the previous symbol, selects the symbol with the highest probability through the probability distribution, and outputs it, gradually generating a complete mathematical formula symbol sequence. The previous hidden layer state uses the previous symbol, takes its feature code and inputs it into the decoder GRU layer to obtain it.

9. The method for dynamic recognition of handwritten mathematical formulas according to claim 7, wherein: In step 4, according to the probability distribution of symbol recognition, combined with the statistical results and confusion matrix, the generated mathematical formula symbol sequence is corrected through LaTeX grammar rules.

Images