Handwritten mathematical formula identification method based on CSP feature fusion

Abstract

The invention discloses a handwritten mathematical formula identification method based on CSP feature fusion, and the method comprises the steps: S1, obtaining image data containing a handwritten mathematical formula, and carrying out the data amplification of the image data to obtain a handwritten mathematical formula image data set; s2, constructing a handwritten mathematical formula recognition network model based on CSP feature fusion; the handwritten mathematical formula recognition network model comprises a feature extraction module, a CSP-based feature fusion module, a multi-scale counting module and an attention decoder module which are connected in sequence; s3, determining a loss function of the handwritten mathematical formula recognition network model, training the handwritten mathematical formula recognition network model according to the obtained handwritten mathematical formula image data set until the loss function converges, and further obtaining a trained handwritten mathematical formula recognition network model, therefore, the handwritten mathematical formula image to be recognized can be recognized. The invention discloses a handwritten mathematical formula identification method. Symbol segmentation is easily influenced by image quality and mathematical formula complexity, accuracy of symbol identification is easily limited by feature extraction and classifier performance, structural analysis is influenced by a hierarchical structure of a mathematical formula, and the number of various symbols is not perceived in the identification process. And the phenomena of missing symbols, mistakenly measuring repeated symbols and the like occur, so that the handwritten mathematical formula is difficult to recognize and poor in precision.

Description

Technical Field

[0001] The present invention relates to the technical field of handwritten mathematical formula recognition, and in particular to a handwritten mathematical formula recognition method based on CSP feature fusion. Background Art

[0002] The existing handwritten mathematical formula recognition (HMER) methods mainly include the following:

[0003] (1) Based on traditional HMER method: Traditional HMER method usually adopts three steps: symbol segmentation, symbol recognition and structure analysis.

[0004] ① Symbol segmentation is to divide the mathematical formulas in the image into individual symbols. This usually involves preprocessing the image and then using edge detection, connected region analysis and other techniques to segment the individual symbols in the mathematical formula.

[0005] ②Symbol recognition is the process of identifying the segmented symbols. Classical classification techniques such as hidden Markov model (HMM), elastic matching and support vector machine (SVM) are usually used to achieve symbol recognition.

[0006] ③ Structural analysis aims to parse the relationship between symbols and organize them into correct mathematical formulas. Context-Free Grammar (CFG), Hidden Markov Model (HMM) and other parsing algorithms are usually used to establish and parse mathematical formulas.

[0007] (2) Deep learning-based methods: Deep learning improves the performance of traditional HMER, and the mainstream framework is the encoder-decoder network.

[0008] ① The encoder network converts the input mathematical formula image into a high-dimensional feature vector or feature sequence, usually using a convolutional neural network (CNN) or a recurrent neural network (RNN) to extract the feature information of the image.

[0009] ②The decoder network receives the feature representation generated by the encoder and outputs the structured result of the mathematical formula by step-by-step decoding. The decoder can be implemented using models such as recurrent neural network (RNN), GRU or Transformer.

[0010] Through the encoder-decoder network architecture, deep learning-based methods can directly learn the mapping relationship from images to mathematical formulas end-to-end, and can learn more complex and abstract feature representations and rules from large amounts of data, thereby achieving more accurate and robust performance in mathematical formula recognition and understanding tasks.

[0011] However, the existing handwritten mathematical formula recognition (HMER) methods have the following problems:

[0012] (1) Based on the traditional HMER method:

[0013] ①Symbol segmentation is easily affected by image quality and the complexity of mathematical formulas. When the input image quality is low or the mathematical formula is complex, the traditional symbol segmentation algorithm will not be able to accurately segment the formula into separate symbols, resulting in errors in the recognition and analysis of subsequent steps.

[0014] ②The accuracy of symbol recognition is easily limited by feature extraction and classifier performance. Traditional HMER methods such as hidden Markov models, elastic matching, and support vector machines and other classic classification technologies perform poorly in complex backgrounds and diverse writing styles, resulting in low accuracy of symbol recognition.

[0015] ③ Structural analysis will be affected by the hierarchical structure of mathematical formulas. Since mathematical formulas have complex hierarchical structures, such as fractions and superscripts and subscripts, the parser cannot accurately understand these structures when identifying and reconstructing them, resulting in inaccurate analysis results.

[0016] (2) Deep learning-based methods: The encoder-decoder architecture is widely used in handwritten mathematical formula recognition. However, due to the lack of perception of the number of various symbols in the recognition process, phenomena such as missing symbols and misdetecting repeated symbols may occur, which shows that current models still face challenges in complex mathematical structures. Summary of the invention

[0017] The present invention provides a handwritten mathematical formula recognition method based on CSP feature fusion to overcome the above technical problems.

[0018] In order to achieve the above object, the technical solution of the present invention is:

[0019] A handwritten mathematical formula recognition method based on CSP feature fusion specifically includes the following steps:

[0020] S1: Acquire image data containing handwritten mathematical formulas, and perform data augmentation on the image data to obtain a handwritten mathematical formula image dataset;

[0021] S2: Construct a handwritten mathematical formula recognition network model based on CSP feature fusion;

[0022] The handwritten mathematical formula recognition network model includes a feature extraction module, a CSP-based feature fusion module, a multi-scale counting module, and an attention decoder module connected in sequence;

[0023] The feature extraction module is used to extract the multi-scale feature map of the image data of the handwritten mathematical formula and transmit it to the feature fusion module based on CSP;

[0024] The CSP-based feature fusion module is used to perform feature fusion on the multi-scale feature map to obtain a fused feature map, and transmit it to the multi-scale counting module and the attention decoder module respectively;

[0025] The multi-scale counting module predicts the category of the mathematical symbol on the input fusion feature map to obtain a feature map count vector, and transmits it to the attention decoder module;

[0026] The attention decoder module is used to obtain the recognition prediction result of the handwritten mathematical formula according to the fused feature map and the feature map count vector;

[0027] S3: Determine the loss function of the handwritten mathematical formula recognition network model, and train the handwritten mathematical formula recognition network model according to the acquired handwritten mathematical formula image data set until the loss function converges, thereby obtaining the trained handwritten mathematical formula recognition network model for realizing the recognition of the handwritten mathematical formula image to be recognized.

[0028] Furthermore, the feature extraction module described in S2 extracts the features of the handwritten mathematical formula from the sample data in the input handwritten mathematical formula image data set.

[0029] It includes a first convolutional layer, a first densely connected convolutional layer, a second convolutional layer, a first pooling layer, a second densely connected convolutional layer, a third convolutional layer, a second pooling layer, a third densely connected convolutional layer, a third pooling layer, and a first linear layer connected in sequence;

[0030] The first convolution layer, the second convolution layer and the third convolution layer are used to extract local pixel features of the handwritten mathematical formula of the input feature map or the feature map input by the previous layer, and obtain the local feature maps corresponding to each other;

[0031] The first densely connected convolutional layer, the second densely connected convolutional layer and the third densely connected convolutional layer are used to extract global pixel features of the handwritten mathematical formula of the feature map input by the previous layer, and obtain the corresponding global feature maps;

[0032] The first pooling layer, the second pooling layer and the third pooling layer are used to perform a pooling operation on the feature map input by the previous level;

[0033] The first linear layer is used to perform feature mapping of its input feature map to the output space;

[0034] The first densely connected convolutional layer outputs a first scale feature map;

[0035] The second densely connected convolutional layer outputs a second scale feature map;

[0036] The output of the first linear layer is used as the third-scale feature map, and finally a multi-scale feature map is obtained.

[0037] Further, the CSP-based feature fusion module described in S2 includes a first feature fusion network layer, a second feature fusion network layer, a third feature fusion network layer, and a fourth feature fusion network layer connected in sequence;

[0038] The first feature fusion network layer includes a first upsampling layer, a first splicing layer and a first CSP network layer connected in sequence;

[0039] The second feature fusion network layer includes a second upsampling layer, a second splicing layer and a second CSP network layer connected in sequence;

[0040] The third feature fusion network layer includes a first downsampling layer, a third splicing layer and a third CSP network layer connected in sequence;

[0041] The fourth feature fusion network layer includes a second downsampling layer, a fourth splicing layer and a fourth CSP network layer connected in sequence;

[0042] The first upsampling layer and the second upsampling layer are used to perform upsampling operations on the input feature map or the feature map input by the previous level;

[0043] The first splicing layer, the second splicing layer, the third splicing layer and the fourth splicing layer are used to perform channel dimension splicing operations on the feature map input by the previous layer;

[0044] The first downsampling layer and the second downsampling layer are used to downsample the feature map input by the previous layer;

[0045] The first CSP network layer, the second CSP network layer, the third CSP network layer and the fourth CSP network layer are used to perform feature fusion operations on the feature graph input by the previous layer;

[0046] And the CSP-based feature fusion module is used to obtain a fused feature map according to the multi-scale feature map;

[0047] That is, the input feature maps of the first upsampling layer and the fourth concatenation layer are both third-scale feature maps;

[0048] The input feature map of the first concatenated layer is the second scale feature map and the output of the first upsampling layer;

[0049] The input feature map of the second concatenation layer is the first scale feature map and the output of the second upsampling layer;

[0050] The first CSP network layer, the second CSP network layer, the third CSP network layer and the fourth CSP network layer output a first feature fusion map, a second feature fusion map, a third feature fusion map and a fourth feature fusion map respectively;

[0051] The input feature map of the third splicing layer is the output of the first feature fusion map and the first downsampling layer; the input feature map of the fourth splicing layer is the output of the third scale feature map and the second downsampling layer;

[0052] The fused feature map output by the feature fusion module based on CSP is the fourth feature fusion map.

[0053] Further, the multi-scale counting module described in S2 includes an input layer, a first module branch, a second module branch, and an output layer;

[0054] The input layer is used to input the fused feature map into the first module branch and the second module branch respectively;

[0055] The first module branch and the second module branch each include a convolution & normalization layer, a channel attention module, a first convolution module, and an addition pooling layer connected in sequence;

[0056] And the convolution kernels of the convolution & normalization layer in the first module branch and the second module branch are different;

[0057] The convolution & normalization layer is used to perform convolution and normalization operations on the fused feature map;

[0058] The channel attention module is used to perform global average pooling on the output of the convolution & normalization layer;

[0059] The first convolution module is used to reduce the number of feature map channels on the deep feature map, and the final number of feature map channels is used as the number of symbol categories of the handwritten mathematical formula;

[0060] The depth feature map is a feature map obtained by performing a pixel-by-pixel multiplication operation on the output of the convolution & normalization layer and the channel attention module;

[0061] The sum pooling layer is used to perform sum pooling operations on the count feature map to obtain the pooling feature map;

[0062] The counting feature map is a feature map for reflecting the symbol category obtained by the Sigmoid function according to the output of the first convolution module;

[0063] The output layer is used to output a feature map count vector;

[0064] And the feature map count vector is the feature map count vector obtained by performing a pixel-by-pixel averaging operation on the pooled feature map output by the first module branch and the pooled feature map output by the second module branch.

[0065] Further, the attention decoder module described in S2 includes a multi-core attention convolution layer, a Transformer network layer, a second convolution module, a first linear layer & Softmax, a second linear layer & Softmax, a second linear layer, a third linear layer, a fourth linear layer, a fifth linear layer, a sixth linear layer, a seventh linear layer and a GRU unit;

[0066] The multi-core attention convolution layer is used to perform a multi-core attention convolution operation on the input fusion feature map to obtain the attention feature map;

[0067] The Transformer network layer is used to perform character position encoding operations on the attention feature map to obtain character position encoding features;

[0068] The second convolution module is used to extract pixel features of the handwritten mathematical formula from the attention feature map, and transmit the output of the second convolution module to the second linear layer;

[0069] The second linear layer is used to perform a feature mapping operation on the output of the second convolution module;

[0070] The GRU unit is used to obtain the hidden layer state feature map according to the character position encoding feature corresponding to the recognition prediction result output by the previous iteration of the second linear layer & Softmax, and transmit it to the third linear layer and the fourth linear layer respectively;

[0071] The third linear layer and the fourth linear layer are respectively used to perform feature mapping operations on the hidden layer state feature map;

[0072] The first linear layer & Softmax is used to obtain the formula character category probability distribution according to the first intermediate feature map to obtain the predicted probability of the character category, and obtain the attention weight for character recognition according to the predicted probability;

[0073] The first intermediate feature map is a feature map obtained by performing a pixel-by-pixel sum operation on the output of the multi-core attention convolutional layer, the output of the Transformer network layer, the output of the second linear layer, and the output of the third linear layer, and processing the output after the pixel-by-pixel sum operation with a Tanh activation function;

[0074] The sixth linear layer is used to perform a feature mapping operation on the context vector;

[0075] And the context vector is obtained by performing a pixel-by-pixel multiplication operation on the output of the first linear layer & Softmax and the fused feature map;

[0076] The seventh linear layer is used to perform a feature mapping operation on the feature map count vector;

[0077] The second linear layer & Softmax is used to obtain the formula character category probability distribution according to the second intermediate feature map to obtain the predicted probability of the character category, and obtain the attention weight for character recognition according to the predicted probability;

[0078] The second intermediate feature map is a feature map obtained by performing a pixel-by-pixel addition operation on the output of the fourth linear layer, the output of the fifth linear layer, the output of the sixth linear layer, and the output of the seventh linear layer;

[0079] According to the second linear layer & Softmax, the symbol with the largest prediction probability of the character category is selected as the recognition prediction result of the handwritten mathematical formula outputted by the current iteration.

[0080] Furthermore, the loss function L of the handwritten mathematical formula recognition network model described in S3 is 总 , whose expression is

[0081]

[0082] Where: p(y t ) represents the symbol y of the predicted t-th step t The probability value after the second linear layer & Softmax; g(y t ) represents the symbol y of the predicted t-th step t The category of ; T represents the number of iterations.

[0083] Furthermore, the method for performing data augmentation on the image data containing the handwritten mathematical formula in S1 is specifically as follows:

[0084] Performing image rotation and image flipping operations on image data containing handwritten mathematical formulas;

[0085] The image rotation includes at least 90 degree rotation, 180 degree rotation and 270 degree rotation.

[0086] Compared with the prior art, the present invention has the following beneficial effects:

[0087] (1) The feature fusion module used is mainly composed of a cross-stage partial (CSP) network. CSP effectively fuses features of different levels and channels, which can improve the network's expressiveness, reduce the number of parameters, and alleviate the gradient vanishing problem, thereby improving the model's robustness and recognition accuracy. This feature fusion method helps the model better capture the feature information in the image, thereby improving the recognition performance of mathematical formulas.

[0088] (2) By combining the multi-core attention convolution layer in the counted attention decoder to perform multi-core attention convolution operations, the attention mechanism is applied to the convolution operation, which enables the model to pay more attention to important areas in the image and extract information and fuse features at multiple scales, thereby improving the recognition accuracy of the model. Compared with traditional convolution operations, multi-core attention convolution operations can more effectively capture key information in the image and help improve recognition performance.

[0089] (3) By augmenting the image data containing handwritten mathematical formulas, that is, augmenting the handwritten mathematical formula image dataset by rotation, flipping, etc., the diversity of the data can be increased, and the generalization ability and recognition accuracy of the model can be improved. Data augmentation can make the model better adapt to handwritten mathematical formulas of different angles, different postures and different styles, thereby improving the robustness of the model and the practical application effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0091] Figure 1 It is a flow chart of the handwritten mathematical formula recognition method based on CSP feature fusion of the present invention;

[0092] Figure 2 It is a flowchart of the offline training phase and the online recognition phase of the handwritten mathematical formula recognition based on CSP feature fusion in this embodiment;

[0093] Figure 3 is a schematic diagram of the result of the image rotation operation in this embodiment;

[0094] Figure 4 This is a schematic diagram of the result of the image flipping operation in this embodiment;

[0095] Figure 5 Schematic diagram of the structure of the feature extraction module in this embodiment;

[0096] Figure 6 This is a schematic diagram of the structure of a cross-stage partial CSP network in this embodiment;

[0097] Figure 7 Schematic diagram of the structure of the feature fusion module based on CSP in this embodiment;

[0098] Figure 8 Schematic diagram of the structure of the multi-scale counting module in this embodiment;

[0099] Fig. 9 Schematic diagram of the structure of the attention decoder module combined with counting in this embodiment. DETAILED DESCRIPTION

[0100] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0101] This embodiment provides a handwritten mathematical formula recognition method based on CSP feature fusion, such as Figure 1 to Figure 2 As shown, the specific steps include:

[0102] S1: Acquire image data containing handwritten mathematical formulas, and perform data augmentation on the image data to obtain a handwritten mathematical formula image dataset;

[0103] Specifically, the method for data augmentation is:

[0104] Performing image rotation and image flipping operations on image data containing handwritten mathematical formulas;

[0105] The image rotation includes at least 90 degree rotation, 180 degree rotation and 270 degree rotation;

[0106] In this embodiment, in order to improve the generalization ability and robustness of the model, data augmentation is performed on the handwritten mathematical formula image in the following manner:

[0107] (1) Read image I and rotate it by 90 degrees, 180 degrees, and 270 degrees respectively to obtain the corresponding rotated images I1, I2, and I3, as shown in Figure 3 As shown:

[0108] (2) Flip the image I left to right to obtain the flipped image I4, as shown in Figure 4 As shown:

[0109] S2: Construct a handwritten mathematical formula recognition network model based on CSP feature fusion;

[0110] The handwritten mathematical formula recognition network model includes a feature extraction module, a CSP-based feature fusion module, a multi-scale counting module, and an attention decoder module connected in sequence;

[0111] The feature extraction module is used to extract the multi-scale feature map of the image data of the handwritten mathematical formula and transmit it to the feature fusion module based on CSP;

[0112] In a specific embodiment, the feature extraction module can perform feature extraction of handwritten mathematical formulas on sample data in an input handwritten mathematical formula image data set, and includes a first convolutional layer, a first densely connected convolutional layer, a second convolutional layer, a first pooling layer, a second densely connected convolutional layer, a third convolutional layer, a second pooling layer, a third densely connected convolutional layer, a third pooling layer, and a first linear layer connected in sequence;

[0113] The first convolution layer, the second convolution layer and the third convolution layer are used to extract local pixel features of the handwritten mathematical formula of the input feature map or the feature map input by the previous layer, and obtain the local feature maps corresponding to each other;

[0114] The first densely connected convolutional layer, the second densely connected convolutional layer and the third densely connected convolutional layer are used to extract global pixel features of the handwritten mathematical formula of the feature map input by the previous layer, and obtain the corresponding global feature maps;

[0115] The first pooling layer, the second pooling layer and the third pooling layer are used to perform a pooling operation on the feature map input by the previous level;

[0116] The first linear layer is used to perform feature mapping of its input feature map to the output space;

[0117] The first densely connected convolutional layer outputs a first scale feature map A;

[0118] The second densely connected convolutional layer outputs a second scale feature map B;

[0119] The first linear layer outputs the third-scale feature map C, and finally obtains a multi-scale feature map.

[0120] like Figure 5As shown, in this embodiment, the best DenseNet is used as the feature extraction module, which is composed of three densely connected convolutional layers (Dense Block), three convolutional layers (Conv), three pooling layers (Pool) and a linear layer (Linear). After the input handwritten mathematical formula image is extracted by the network, three feature maps of different scales are obtained: feature map A (reduced by 8 times relative to the original size), feature map B (reduced by 16 times relative to the original size) and feature map C (reduced by 32 times relative to the original size).

[0121] The CSP-based feature fusion module is used to perform feature fusion on the multi-scale feature map to obtain a fused feature map, and transmit it to the multi-scale counting module and the attention decoder module respectively;

[0122] In a specific embodiment, the CSP-based feature fusion module includes a first feature fusion network layer, a second feature fusion network layer, a third feature fusion network layer, and a fourth feature fusion network layer connected in sequence;

[0123] The first feature fusion network layer includes a first upsampling layer, a first splicing layer and a first CSP network layer connected in sequence;

[0124] The second feature fusion network layer includes a second upsampling layer, a second splicing layer and a second CSP network layer connected in sequence;

[0125] The third feature fusion network layer includes a first downsampling layer, a third splicing layer and a third CSP network layer connected in sequence;

[0126] The fourth feature fusion network layer includes a second downsampling layer, a fourth splicing layer and a fourth CSP network layer connected in sequence;

[0127] The first upsampling layer and the second upsampling layer are used to perform upsampling operations on the input feature map or the feature map input by the previous level;

[0128] The first splicing layer, the second splicing layer, the third splicing layer and the fourth splicing layer are used to perform channel dimension splicing operations on the feature map input by the previous layer;

[0129] The first downsampling layer and the second downsampling layer are used to downsample the feature map input by the previous layer;

[0130] The first CSP network layer, the second CSP network layer, the third CSP network layer and the fourth CSP network layer are used to perform feature fusion operations on the feature graph input by the previous layer;

[0131] And the CSP-based feature fusion module is used to obtain a fused feature map according to the multi-scale feature map;

[0132] That is, the input feature maps of the first upsampling layer and the fourth concatenation layer are both third-scale feature maps;

[0133] The input feature map of the first concatenated layer is the second scale feature map and the output of the first upsampling layer;

[0134] The input feature map of the second concatenation layer is the first scale feature map and the output of the second upsampling layer;

[0135] The first CSP network layer, the second CSP network layer, the third CSP network layer and the fourth CSP network layer respectively output a first feature fusion graph B′, a second feature fusion graph A′, a third feature fusion graph C′ and a fourth feature fusion graph F;

[0136] The input feature map of the third splicing layer is the first feature fusion map B′ and the output of the first downsampling layer; the input feature map of the fourth splicing layer is the third scale feature map C and the output of the second downsampling layer;

[0137] The fused feature map output by the feature fusion module based on CSP is the fourth feature fusion map F.

[0138] like Figure 6As shown, the feature fusion module based on CSP in this embodiment is mainly composed of a cross-stage partial (Cross Stage Partial, CSP) network. CSP is a technology used to enhance network performance in the field of computer vision. It mainly improves the performance of the model through staged feature segmentation and fusion. And the network structures of the first CSP network layer, the second CSP network layer, the third CSP network layer and the fourth CSP network layer in the first feature fusion network layer, the second feature fusion network layer, the third feature fusion network layer and the fourth feature fusion network layer are all composed of two Conv2D_BN_SiLU modules, a channel separation module and three Bottleneck modules, wherein the Conv2D_BN_SiLU module is a standard convolution unit, including a convolution layer (Conv2D), batch normalization (Batch Normalization) and SiLU activation function. Through the 1×1 convolution layer, the number of channels is reduced by half. Batch normalization ensures the stability of the feature map, making subsequent processing more efficient. The SiLU activation function introduces nonlinearity, which enhances the network's ability to capture complex features and improves the expression of the final features. Finally, this module is used to reduce the dimension of the feature map to facilitate subsequent operations; the channel separation module divides the input feature map into two parts in the channel dimension, usually directly passing any part of the reduced feature map through a shallow path (without processing), and the other part is further processed through a deep path (three Bottleneck operations); the Bottleneck module is a common bottleneck structure module used to reduce the amount of calculation and the number of parameters while maintaining network performance.

[0139] The feature fusion module based on CSP is shown in the attached Figure 7 As shown in Figure 1, it fuses the three feature maps of different scales extracted by the feature extraction module. The fusion process starts from the feature map C. This is a bottom-up process, and its steps are as follows:

[0140] After upsampling, feature map C is concatenated with feature map B, which is then input into the CSP network for feature fusion to obtain feature map B′; the fused feature map B′ is upsampled and concatenated with feature map A, which is then input into the CSP network for feature fusion to obtain feature map A′; feature map A′ is downsampled and concatenated with feature map B′, which is then input into the CSP network for feature fusion to obtain feature map C′; feature map C′ is downsampled and concatenated with feature map C, which is then fused in the CSP network, and the final feature map F with a size of H×W×684 is output.

[0141] The multi-scale counting module predicts the category of the mathematical symbol on the input fusion feature map to obtain a feature map count vector, and transmits it to the attention decoder module;

[0142] In a specific embodiment, the multi-scale counting module includes an input layer, a first module branch, a second module branch, and an output layer;

[0143] The input layer is used to input the fused feature map into the first module branch and the second module branch respectively;

[0144] The first module branch and the second module branch each include a convolution & normalization layer, a channel attention module, a first convolution module, and an addition pooling layer connected in sequence;

[0145] And the convolution kernels of the convolution & normalization layer in the first module branch and the second module branch are different;

[0146] The convolution & normalization layer is used to perform convolution and normalization operations on the fused feature map;

[0147] The channel attention module is used to perform global average pooling on the output of the convolution & normalization layer;

[0148] The first convolution module is used to reduce the number of feature map channels on the deep feature map S, and use the final number of feature map channels as the number of symbol categories of the handwritten mathematical formula;

[0149] The depth feature map S is a feature map obtained by performing a pixel-by-pixel multiplication operation on the output of the convolution & normalization layer and the channel attention module;

[0150] The sum pooling layer is used to perform sum pooling operations on the count feature map M to obtain a pooling feature map;

[0151] The counting feature map M is a feature map for reflecting the symbol category obtained by the Sigmoid function according to the output of the first convolution module;

[0152] The output layer is used to output a feature map count vector ν;

[0153] And the feature map count vector is the feature map count vector ν obtained by performing a pixel-by-pixel averaging operation on the pooled feature map output by the first module branch and the pooled feature map output by the second module branch.

[0154] The multi-scale counting module in this embodiment can predict the category of each symbol according to the input feature map and generate a one-dimensional counting vector ν. The module has two parallel branches with different sizes of convolution kernels (3×3 and 55) for extracting multi-scale features of the formula image, such as Figure 8 As shown, one of the module branches is used for illustration:

[0155] 1) Perform convolution and normalization operations on the feature map F of size H×W×684 to obtain the feature map F′∈R H×W×C′ , where C′ is the number of feature map channels (the number of channels is the number of representation categories), which is input into the channel attention module (the channel attention module consists of an average pooling layer, two linear layers, a ReLU activation function and a Sigmoid activation function), and the feature map F″ is obtained through the global average pooling layer in the channel attention module.

[0156] 2) Perform pixel-by-pixel point multiplication on the feature map F′ and the feature map F″ to increase the expressive power of the model and obtain the feature map S. The mathematical description of this process is:

[0157] F″=σ(W1(G(F′))+b1)

[0158]

[0159] Where: G represents global average pooling; σ and g represent ReLU function and Sigmoid function respectively. represents pixel-by-pixel multiplication; W1, W2, b1, b2 represent trainable weights.

[0160] 3) Using a 1×1 convolution module, the number of channels of the deep feature map S is reduced from C′ to C, where C is the number of symbol categories, and the Sigmoid function is used to generate a count feature map M in the range of (0, 1). Each M i ∈R H×W×C Reflects the position of the i-th symbol class.

[0161] 4) Count feature map M i Perform summing and pooling operations on the H and W dimensions to obtain the count vector ν∈R 1×C , which is mathematically described as:

[0162]

[0163] Where: ν i ∈R 1×1 Represents the predicted count of the i-th character.

[0164] 5) Since the feature maps of the two module branches of the 3×3 convolution kernel and the 5×5 convolution kernel contain information of different scales and are highly complementary to each other, these complementary count vectors are combined and the final result ν∈R is generated using a pixel-by-pixel averaging operation 1×C , which is then fed into the combined computational attention decoder (CCAD).

[0165] The attention decoder module is used to obtain the recognition prediction result of the handwritten mathematical formula according to the fused feature map and the feature map count vector;

[0166] In a specific embodiment, the attention decoder module includes a multi-core attention convolution layer, a Transformer network layer, a second convolution module, a first linear layer & Softmax, a second linear layer & Softmax, a second linear layer, a third linear layer, a fourth linear layer, a fifth linear layer, a sixth linear layer, a seventh linear layer and a GRU unit;

[0167] The multi-core attention convolution layer is used to perform a multi-core attention convolution operation on the fusion feature map to obtain the attention feature map;

[0168] The Transformer network layer is used to perform character position encoding operations on the acquired attention feature map to obtain character position encoding features;

[0169] The second convolution module is used to extract pixel features of the handwritten mathematical formula from the attention feature map, and transmit the output of the second convolution module to the second linear layer;

[0170] The second linear layer is used to perform a feature mapping operation on the output of the second convolution module;

[0171] The GRU unit is used to obtain the hidden layer state feature map according to the character position encoding feature corresponding to the recognition prediction result output by the previous iteration of the second linear layer & Softmax, and transmit it to the third linear layer and the fourth linear layer respectively;

[0172] The third linear layer and the fourth linear layer are respectively used to perform feature mapping operations on the hidden layer state feature map;

[0173] The first linear layer & Softmax is used to obtain the formula character category probability distribution according to the first intermediate feature map to obtain the predicted probability of the character category, and obtain the attention weight for character recognition according to the predicted probability;

[0174] The first intermediate feature map is a feature map obtained by performing a pixel-by-pixel sum operation on the output of the multi-core attention convolutional layer, the output of the Transformer network layer, the output of the second linear layer, and the output of the third linear layer, and processing the output after the pixel-by-pixel sum operation with a Tanh activation function;

[0175] The sixth linear layer is used to perform a feature mapping operation on the context vector;

[0176] And the context vector is obtained by performing a pixel-by-pixel multiplication operation on the output of the first linear layer & Softmax and the fused feature map;

[0177] The seventh linear layer is used to perform a feature mapping operation on the feature map count vector;

[0178] The second linear layer & Softmax is used to obtain the formula character category probability distribution according to the second intermediate feature map to obtain the predicted probability of the character category, and obtain the attention weight for character recognition according to the predicted probability;

[0179] The second intermediate feature map is a feature map obtained by performing a pixel-by-pixel addition operation on the output of the fourth linear layer, the output of the fifth linear layer, the output of the sixth linear layer, and the output of the seventh linear layer;

[0180] According to the second linear layer & Softmax, the symbol with the largest prediction probability of the character category is selected as the recognition prediction result of the handwritten mathematical formula outputted by the current iteration.

[0181] In this embodiment, the attention decoder module combined with the count is H×W×684 and the count vector ν∈R output by the multi-scale counting module 1×C The purpose of symbol prediction is to introduce symbol counting results as additional global information to improve recognition accuracy. Each linear layer in the module is used to map features to the output space, and the linear layer & Softmax is used to convert the output of the linear layer into a probability distribution, thereby providing the predicted probability of each category. The prediction process is shown in the attached figure. Fig. 9 As shown, the specific steps are as follows:

[0182] 1) For the feature map F∈R H×W×684 Perform multi-core attention convolution operations to more effectively capture the key information in the image and obtain the feature map T∈R H×W×512 .

[0183] 2) In order to improve the model’s perception of spatial position, the feature is positionally encoded using Transformer to obtain the position encoding P of the feature map.

[0184] 3) In order to avoid repeatedly paying attention to a certain area or missing a certain area, the feature map of size H×W is processed by an 11×11 convolution layer, i.e., the second convolution module, to obtain the coverage attention A, which means the sum of all past weights.

[0185] 4) In the decoding step t, the symbol y obtained by decoding in the previous step is t-1 Take its feature code E(y t-1 )∈R 1×256 Input into the GRU unit to get its hidden layer state h t ∈R 1×256 .

[0186] 5) Combine the feature map T, position encoding P, coverage attention A and hidden layer state h t ∈R1×256 , calculate the current attention weight, which is mathematically described as:

[0187] e t =ω T tanh(T+P+W a A+W h h t )+b

[0188] Where: e t represents the attention score at time step t;

[0189]

[0190] Where: e t,ij represents the score value of the attention weight between the i-th target symbol and the j-th source symbol at time step t; e t,pq represents the attention weight score between the p-th target symbol and the q-th source symbol at time step t; α t，ij represents the current attention weight, ω,b,W a ,W h Represents trainable weights.

[0191] 6) Set the attention weight α t Multiply pixel by pixel with the feature map F to get the context vector L∈R 1×256 .

[0192] 7) In order to prevent prediction errors caused by symbol omission, the count vector ν, the context vector L, and the hidden layer state h are combined t And the feature code E(y t1 ) is decoded, and its mathematical description is:

[0193]

[0194] y t ~p(y t )

[0195] Where: o ,b o ,W c ,W v ,W t ,W e Represents trainable weights.

[0196] 8) The final symbol prediction output y t Depends on the probability value p(y t ), that is, take the symbol with the highest probability as the output.

[0197] S3: Determine the loss function of the handwritten mathematical formula recognition network model, and train the handwritten mathematical formula recognition network model according to the acquired handwritten mathematical formula image data set until the loss function converges, thereby obtaining the trained handwritten mathematical formula recognition network model for realizing the recognition of the handwritten mathematical formula image to be recognized.

[0198] In this embodiment, the handwritten mathematical formula image after data augmentation is input into the handwritten mathematical formula recognition network based on CSP feature fusion, and the handwritten mathematical formula recognition model is trained offline. The training set of the handwritten mathematical formula image is the public dataset CROHME, and the test set is CROHME 2014, CROHME 2016 and CROHME 2019;

[0199] The public data set CROHME is used for training. The training set contains 8836 handwritten mathematical formulas. In addition, the three test sets contain 986, 1147 and 1199 handwritten mathematical formulas respectively. The number of symbol classes contained in CROHME is 112, including the characters "sos" and "eos". This embodiment adopts the cross entropy loss function. The same optimization process is used for the decoder that decodes from left to right and the decoder that decodes from right to left. The loss function L of the handwritten mathematical formula recognition network model is 总 , whose expression is

[0200]

[0201] Where: p(y t ) represents the symbol y of the predicted t-th step t The probability value after the second linear layer & Softmax; g(y t ) represents the symbol y of the predicted t-th step t T represents the number of iterations. Full training is used, with a total of 240 rounds, a batch size of 32, and an optimizer of Adadelta.

[0202] In summary, the handwritten mathematical formula recognition method based on CSP feature fusion disclosed in this embodiment has the following beneficial effects:

[0203] (1) The feature fusion module used in this embodiment is mainly composed of a cross-stage partial (CSP) network. By effectively fusing features of different levels and channels through CSP, the network's expressiveness can be improved, the number of parameters can be reduced, and the gradient vanishing problem can be alleviated, thereby improving the robustness and recognition accuracy of the model. This feature fusion method helps the model better capture the feature information in the image, thereby improving the recognition performance of the mathematical formula.

[0204] (2) In this embodiment, the 1×1 convolution operation in the attention decoder combined with the count is replaced by a multi-core attention convolution operation. The attention mechanism is applied to the convolution operation, which can make the model pay more attention to the important areas in the image and perform information extraction and feature fusion at multiple scales, thereby improving the recognition accuracy of the model. Compared with the traditional convolution operation, the multi-core attention convolution operation can more effectively capture the key information in the image, which helps to improve the recognition performance.

[0205] (3) In this embodiment, the handwritten mathematical formula image dataset is augmented by rotation, flipping, etc., which can increase the diversity of the data and improve the generalization ability and recognition accuracy of the model. Data augmentation can make the model better adapt to handwritten mathematical formulas of different angles, different postures and different styles, thereby improving the robustness of the model and the practical application effect.

[0206] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A handwritten mathematical formula recognition method based on CSP feature fusion, characterized in that: The specific steps include: S1: Acquire image data containing handwritten mathematical formulas, and perform data augmentation on the image data to obtain a handwritten mathematical formula image dataset; S2: Construct a handwritten mathematical formula recognition network model based on CSP feature fusion; The handwritten mathematical formula recognition network model includes a feature extraction module, a CSP-based feature fusion module, a multi-scale counting module, and an attention decoder module connected in sequence; The feature extraction module is used to extract the multi-scale feature map of the image data of the handwritten mathematical formula and transmit it to the feature fusion module based on CSP; The CSP-based feature fusion module is used to perform feature fusion on the multi-scale feature map to obtain a fused feature map, and transmit it to the multi-scale counting module and the attention decoder module respectively; The multi-scale counting module predicts the category of the mathematical symbol on the input fusion feature map to obtain a feature map count vector, and transmits it to the attention decoder module; The attention decoder module is used to obtain the recognition prediction result of the handwritten mathematical formula according to the fused feature map and the feature map count vector; S3: Determine the loss function of the handwritten mathematical formula recognition network model, and train the handwritten mathematical formula recognition network model according to the acquired handwritten mathematical formula image data set until the loss function converges, thereby obtaining the trained handwritten mathematical formula recognition network model for realizing the recognition of the handwritten mathematical formula image to be recognized.

2. According to the method for handwritten mathematical formula recognition based on CSP feature fusion according to claim 1, it is characterized in that: The feature extraction module described in S2 can extract the features of handwritten mathematical formulas from the sample data in the input handwritten mathematical formula image data set. It includes a first convolutional layer, a first densely connected convolutional layer, a second convolutional layer, a first pooling layer, a second densely connected convolutional layer, a third convolutional layer, a second pooling layer, a third densely connected convolutional layer, a third pooling layer, and a first linear layer connected in sequence; The first convolution layer, the second convolution layer and the third convolution layer are used to extract local pixel features of the handwritten mathematical formula of the input feature map or the feature map input by the previous layer, and obtain the local feature maps corresponding to each other; The first densely connected convolutional layer, the second densely connected convolutional layer and the third densely connected convolutional layer are used to extract global pixel features of the handwritten mathematical formula of the feature map input by the previous layer, and obtain the corresponding global feature maps; The first pooling layer, the second pooling layer and the third pooling layer are used to perform a pooling operation on the feature map input by the previous level; The first linear layer is used to perform feature mapping of its input feature map to the output space; The first densely connected convolutional layer outputs a first scale feature map; The second densely connected convolutional layer outputs a second scale feature map; The first linear layer outputs the third-scale feature map, and finally obtains a multi-scale feature map.

3. According to the method for handwritten mathematical formula recognition based on CSP feature fusion according to claim 2, it is characterized in that: The CSP-based feature fusion module in S2 includes a first feature fusion network layer, a second feature fusion network layer, a third feature fusion network layer, and a fourth feature fusion network layer connected in sequence; The first feature fusion network layer includes a first upsampling layer, a first splicing layer and a first CSP network layer connected in sequence; The second feature fusion network layer includes a second upsampling layer, a second splicing layer and a second CSP network layer connected in sequence; The third feature fusion network layer includes a first downsampling layer, a third splicing layer and a third CSP network layer connected in sequence; The fourth feature fusion network layer includes a second downsampling layer, a fourth splicing layer and a fourth CSP network layer connected in sequence; The first upsampling layer and the second upsampling layer are used to perform upsampling operations on the input feature map or the feature map input by the previous level; The first splicing layer, the second splicing layer, the third splicing layer and the fourth splicing layer are used to perform channel dimension splicing operations on the feature map input by the previous layer; The first downsampling layer and the second downsampling layer are used to downsample the feature map input by the previous layer; The first CSP network layer, the second CSP network layer, the third CSP network layer and the fourth CSP network layer are used to perform feature fusion operations on the feature graph input by the previous layer; And the CSP-based feature fusion module is used to obtain a fused feature map according to the multi-scale feature map; That is, the input feature maps of the first upsampling layer and the fourth concatenation layer are both third-scale feature maps; The input feature map of the first concatenated layer is the second scale feature map and the output of the first upsampling layer; The input feature map of the second concatenation layer is the first scale feature map and the output of the second upsampling layer; The first CSP network layer, the second CSP network layer, the third CSP network layer and the fourth CSP network layer output a first feature fusion map, a second feature fusion map, a third feature fusion map and a fourth feature fusion map respectively; The input feature map of the third splicing layer is the output of the first feature fusion map and the first downsampling layer; the input feature map of the fourth splicing layer is the output of the third scale feature map and the second downsampling layer; The fused feature map output by the feature fusion module based on CSP is the fourth feature fusion map.

4. According to the method of handwritten mathematical formula recognition based on CSP feature fusion according to claim 3, it is characterized in that: The multi-scale counting module described in S2 includes an input layer, a first module branch, a second module branch, and an output layer; The input layer is used to input the fused feature map into the first module branch and the second module branch respectively; The first module branch and the second module branch each include a convolution & normalization layer, a channel attention module, a first convolution module, and an addition pooling layer connected in sequence; And the convolution kernels of the convolution & normalization layer in the first module branch and the second module branch are different; The convolution & normalization layer is used to perform convolution and normalization operations on the fused feature map; The channel attention module is used to perform global average pooling on the output of the convolution & normalization layer; The first convolution module is used to reduce the number of feature map channels on the deep feature map, and the final number of feature map channels is used as the number of symbol categories of the handwritten mathematical formula; The depth feature map is a feature map obtained by performing a pixel-by-pixel multiplication operation on the output of the convolution & normalization layer and the channel attention module; The sum pooling layer is used to perform sum pooling operations on the count feature map to obtain the pooling feature map; The counting feature map is a feature map for reflecting the symbol category obtained by the Sigmoid function according to the output of the first convolution module; The output layer is used to output a feature map count vector; And the feature map count vector is the feature map count vector obtained by performing a pixel-by-pixel averaging operation on the pooled feature map output by the first module branch and the pooled feature map output by the second module branch.

5. According to the method for handwritten mathematical formula recognition based on CSP feature fusion according to claim 4, it is characterized in that: The attention decoder module described in S2 includes a multi-core attention convolution layer, a Transformer network layer, a second convolution module, a first linear layer & Softmax, a second linear layer & Softmax, a second linear layer, a third linear layer, a fourth linear layer, a fifth linear layer, a sixth linear layer, a seventh linear layer and a GRU unit; The multi-core attention convolution layer is used to perform a multi-core attention convolution operation on the input fusion feature map to obtain the attention feature map; The Transformer network layer is used to perform character position encoding operations on the attention feature map to obtain character position encoding features; The second convolution module is used to extract pixel features of the handwritten mathematical formula from the attention feature map, and transmit the output of the second convolution module to the second linear layer; The second linear layer is used to perform a feature mapping operation on the output of the second convolution module; The GRU unit is used to obtain the hidden layer state feature map according to the character position encoding feature corresponding to the recognition prediction result output by the previous iteration of the second linear layer & Softmax, and transmit it to the third linear layer and the fourth linear layer respectively; The third linear layer and the fourth linear layer are respectively used to perform feature mapping operations on the hidden layer state feature map; The first linear layer & Softmax is used to obtain the formula character category probability distribution according to the first intermediate feature map to obtain the predicted probability of the character category, and obtain the attention weight for character recognition according to the predicted probability; The first intermediate feature map is a feature map obtained by performing a pixel-by-pixel sum operation on the output of the multi-core attention convolutional layer, the output of the Transformer network layer, the output of the second linear layer, and the output of the third linear layer, and processing the output after the pixel-by-pixel sum operation with a Tanh activation function; The sixth linear layer is used to perform a feature mapping operation on the context vector; And the context vector is obtained by performing a pixel-by-pixel multiplication operation on the output of the first linear layer & Softmax and the fused feature map; The seventh linear layer is used to perform a feature mapping operation on the feature map count vector; The second linear layer & Softmax is used to obtain the formula character category probability distribution according to the second intermediate feature map to obtain the predicted probability of the character category, and obtain the attention weight for character recognition according to the predicted probability; The second intermediate feature map is a feature map obtained by performing a pixel-by-pixel addition operation on the output of the fourth linear layer, the output of the fifth linear layer, the output of the sixth linear layer, and the output of the seventh linear layer; According to the second linear layer & Softmax, the symbol with the largest prediction probability of the character category is selected as the recognition prediction result of the handwritten mathematical formula outputted by the current iteration.

6. The handwritten mathematical formula recognition method based on CSP feature fusion according to claim 1 is characterized in that: The loss function L of the handwritten mathematical formula recognition network model described in S3 总 , whose expression is Where: p(y t ) represents the symbol y of the predicted t-th step t The probability value after the second linear layer & Softmax; g(y t ) represents the symbol y of the predicted t-th step t The category of ; T represents the number of iterations.

7. The method for handwritten mathematical formula recognition based on CSP feature fusion according to claim 1 is characterized in that: The method for performing data augmentation on image data containing handwritten mathematical formulas in S1 is specifically as follows: Performing image rotation and image flipping operations on image data containing handwritten mathematical formulas; The image rotation includes at least 90 degree rotation, 180 degree rotation and 270 degree rotation.

Images