Freehand sketch recognition method based on multi-level stroke sequence feature extraction

Abstract

The invention discloses a freehand sketch recognition method based on multi-level stroke sequence feature extraction, and the method comprises the steps: obtaining a handwritten formula image sample through a tablet device, and recording stroke track data; secondly, preprocessing the data by utilizing a spatial variant character data enhancement (SCDE) method to improve the retrieval capability of the model; then, a CPCA method is adopted to carry out channel and spatial dimension attention correction on the feature map of the convolutional neural network so as to enhance the feature extraction effect; and finally, decoding the image features into character vectors by using a GRU-based character decoder, and generating a final recognition result by calculating a multi-character loss function. According to the method, the recognition precision of handwritten mathematical formulas is improved, and particularly, the method has excellent performance when coping with handwritten formulas with sparse and complex structures. In addition, space channel attention perceived by characters is further utilized, and the generalization ability and the real-time performance of the model are effectively improved.

Description

Technical Field

[0001] The invention belongs to image data processing technology, and in particular relates to a hand-drawn sketch recognition method based on multi-level stroke sequence feature extraction. Background Art

[0002] Handwritten formulas are a form of expression widely used in mathematics, physics, chemistry and other disciplines. They are intuitive and convenient. Whether in education, scientific research or engineering, handwritten formulas are an important communication tool. However, due to the individual differences in writing, the complexity of expression and the diversity of symbols, traditional character recognition technology faces great challenges in processing handwritten formulas.

[0003] Different writers have significant differences in symbol shape, proportion, arrangement, etc., which makes it difficult to effectively apply traditional template matching or rule-based recognition methods. In addition, handwritten formulas are usually embedded in a sparse background and accompanied by irregular lines, strokes and noise, which further increases the difficulty of recognition. Deep learning can effectively process complex handwritten formula data without relying on manually designed features by automatically extracting features and constructing complex mapping relationships. However, how to use deep learning technology to accurately extract effective information of formulas from sparse backgrounds and improve the accuracy and efficiency of handwritten formula recognition is still a topic with important academic and engineering value. Summary of the invention

[0004] In order to solve the deficiencies of the prior art and achieve the purpose of improving the accuracy and efficiency of handwritten formula recognition, the present invention adopts the following technical solutions:

[0005] The hand-drawn sketch recognition method based on multi-level stroke sequence feature extraction includes the following steps:

[0006] Step 1: Get a hand-drawn sketch;

[0007] Step 2: Perform data augmentation on the hand-drawn sketch;

[0008] Step 3: Construct a pluggable character-aware spatial channel attention mechanism. For the extracted hand-drawn sketch features, adaptively sequence the attention feature map information in the channel and space dimensions based on any convolutional neural network to obtain a new feature map.

[0009] Step 4: Use a recurrent neural network character decoder to decode the new image features into character vectors, calculate the multi-character loss, and generate the final prediction.

[0010] Furthermore, in step 3, the extracted hand-drawn sketch features are input into the channel attention module, and a pooling operation based on the width and height of the feature map is performed. The pooled features are sent to the neural network, and channel attention features are generated based on the features output by the neural network and the hand-drawn sketch features. The channel attention features are input into the spatial attention module, and a channel-based pooling operation is performed to generate spatial attention features. Based on the channel attention features and the spatial attention features, the new feature map is generated. The channel attention module and the spatial attention module can each effectively improve the recognition of the feature map, but through their sequential combination, experiments show that the overall performance of the model far exceeds the effect of applying the two separately. In the recognition task of complex mathematical formulas, the accuracy is improved by more than 15%.

[0011] Furthermore, the channel attention module takes the input hand-drawn sketch feature map F∈R {H*W*C} After global maximum pooling and global average pooling based on width and height respectively, two 1×1×C feature maps F are obtained. avg and F max , where C represents the number of channels; and a pooling parameter α is added to dynamically adjust the pooling weight. The formula after pooling is:

[0012] F pool =α*F avg +(1-α)·F max

[0013] Then, the obtained F pool They are sent to a two-layer neural network, the number of neurons in the first layer is C / r, r represents the reduction rate, the activation function is Relu, and the number of neurons in the second layer is C; the features output by MLP are added and activated by Sigmoid to generate feature W c =σ(MLP(F pool ));

[0014] Then, the channel self-attention feature F ′ =W c ⊙F Input spatial attention module.

[0015] Furthermore, the spatial attention module performs channel-based global maximum pooling and global average pooling on the channel self-attention features to obtain two H×W×1 feature maps. After the channel splicing operation is performed on the two feature maps, a 7×7 convolution operation is performed to reduce the dimension to H×W×1, and a sigmoid activation operation is performed to generate a spatial attention feature, that is, Ms=σ(Conv7×7([F avg ,F max])).

[0016] Furthermore, in step 3, the spatial attention feature M s and channel self-attention feature F ′ Do multiplication to get the final generated new feature F out =M s ⊙F ′ , ⊙ represents the main element multiplication.

[0017] Furthermore, in step 4, for a given handwritten image feature map, high-dimensional image features are first obtained through a feature extraction network, and then feature decoding is performed using a gated recurrent unit (GRU) in a recurrent neural network. The GRU controls the update of the hidden state and the influence of the previous state, and the high-dimensional features are input into the GRU in a time series.

[0018] Furthermore, the GRU receives the output y of the previous time step at each time point t-1 and the hidden state h t-1 , calculate the new hidden state h by updating the formula t :

[0019] h t =z t ⊙h t-1 +(1-z t )⊙h t

[0020] Then, the output is predicted based on the current hidden state through a fully connected layer:

[0021] y t =softmax(Wyh t +b y )

[0022] In this process, a character sequence is generated step by step until the end symbol "eos" is encountered; the entire decoding process optimizes the model by calculating the multi-character cross entropy loss. This loss function measures the gap between the predicted output and the true label, and finally realizes the conversion from image features to character sequences.

[0023] Furthermore, in step 1, the stroke trajectory coordinates are obtained through the hand-drawn sketch to construct a stroke data set corresponding to the hand-drawn sketch, the trajectory information of each stroke includes a starting point and an end point, and the stroke data of the unit sketch are unified into a matrix with a fixed dimension.

[0024] Furthermore, in the step 2, spatially deformable character data enhancement is used to expand the stroke data set, including: performing linear transformation on the hand-drawn sketch, including random rotation and translation, stretching the hand-drawn sketch, simulating the effect of observing the same object from different perspectives, making the image look similar to being taken from different perspectives, and simulating the breakage or thinning of the strokes for the handwriting recognition formula to enhance the model's recognition ability for incomplete or rough writing, including using a 3*3 kernel for corrosion operations, and simulating the bolding of the strokes or the spread of ink for the handwriting recognition formula to enhance the model's adaptability to bold writing or excessive writing, including using a 3x3 kernel for expansion operations.

[0025] A hand-drawn sketch recognition method based on multi-level stroke sequence feature extraction is applied to hand-drawn formula recognition. In step 1, the stroke data of each formula is unified into a matrix with a fixed dimension; in step 3, since the image features of handwritten mathematical formulas are sparse, a densely connected convolutional network is used to improve the performance and generalization ability of the model.

[0026] The advantages and beneficial effects of the present invention are:

[0027] The present invention improves the recognition accuracy of handwritten mathematical formulas by enhancing the retrieval and attention mechanisms, especially when dealing with sparse and complex handwritten formulas. In addition, the present invention also utilizes the spatial channel attention of character perception, effectively improving the generalization ability and real-time performance of the model, and is suitable for scenarios such as intelligent education and formula entry. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a flow chart of the method in an embodiment of the present invention.

[0029] Figure 2 It is a diagram of the architecture of spatially deformable character data enhancement (SCDE) in an embodiment of the present invention.

[0030] Figure 3 It is a pluggable character-aware spatial channel attention (CPCA) architecture diagram in an embodiment of the present invention. DETAILED DESCRIPTION

[0031] The specific implementation of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the present invention, and is not used to limit the present invention.

[0032] like Figure 1 As shown in FIG, a hand-drawn sketch recognition method based on multi-level stroke sequence feature extraction adopts the public dataset CROHME dataset, including the following steps:

[0033] Step 1: Obtain hand-drawn formula images. The hand-written formula samples are obtained in batches by recording the hand-drawn formula images of the writer on the tablet, and the trajectory coordinates of the handwritten strokes are recorded at the same time to obtain the final hand-drawn formula stroke dataset.

[0034] By recording the writer's operation of hand-drawn formula images on a tablet device, batches of handwritten formula sample data are obtained, and the trajectory coordinates of the handwritten strokes are recorded in real time. The final hand-drawn formula stroke sequence data set is generated using these trajectory coordinate data. The trajectory information of each stroke includes parameters such as the starting point and the end point, and the stroke data of each formula is uniformly saved in a matrix format with a fixed dimension.

[0035] Step 2: Spatial Character Data Enhancement: The Spatial Character Data Enhancement (SCDE) method is used to enhance the retrieval capability, and the dataset is randomly processed before training.

[0036] like Figure 2 As shown in the figure, before the model learns, there is a 20% probability of performing one or more of the following four operations on each image, or there is a possibility that no operation is performed, thereby greatly improving the learning ability of the model:

[0037] 1) Perform linear transformation on the image, including random rotation of -5 to 5 degrees and random translation in the horizontal and vertical directions, with a maximum translation of 10% of the image width and height;

[0038] 2) Stretching the image at four corners to simulate the effect of observing the same object from different perspectives, making the image look like it was taken from different perspectives;

[0039] 3) For handwriting recognition formulas, the breaking or thinning of strokes can be simulated to enhance the model's recognition ability for incomplete or rough writing, including using a 3*3 kernel for erosion operations;

[0040] 4) The handwriting recognition formula can simulate the thickening of strokes or the diffusion of ink to enhance the model's adaptability to bold or excessive writing, including the use of a 3x3 kernel for dilation operations.

[0041] Step 3: Character-Aware Spatial Channel Attention Method: A pluggable character-aware spatial channel attention (CPCA) method is used to adaptively and sequentially modify the attention feature map information in the channel and spatial dimensions based on any convolutional neural network to generate a new feature map.

[0042] like Figure 3As shown in the figure, since the features of handwritten mathematical formula images are sparse, a densely connected convolutional network is used to improve the performance and generalization ability of the model. The input feature map F is subjected to global maximum pooling and global average pooling based on width and height respectively to obtain two 1×1×C feature maps. Then, they are sent to a two-layer neural network, the number of neurons in the first layer is C / reduction (reduction is the reduction rate), the activation function is Relu, and the number of neurons in the second layer is C. Then, the features output by the MLP are added, and then the sigmoid activation operation is performed to generate the final channel attention feature. The feature map output by the channel attention module is then used as the input feature map of the next module. Perform a channel-based global max pooling and global average pooling to obtain two H×W×1 feature maps, and then perform channel splicing operations on these two feature maps based on channels. Then, after a 7×7 convolution operation, the dimension is reduced to 1 channel, that is, H×W×1. Then, the spatial attention feature, that is, Ms, is generated through sigmoid. Finally, the feature is multiplied by the input feature of the module to obtain the final generated feature.

[0043] Step 4: Recurrent Neural Network Character Decoder: A recurrent neural network-based character decoder is used to decode the image features into character vectors, calculate the multi-character loss function, and generate the final prediction.

[0044] For the feature map of a given handwritten image, the GRU in the recurrent neural network is used for feature decoding, and the high-dimensional features are embedded in the GRU unit in time series. t-1 , obtain the hidden state of the recurrent neural network in turn, and then predict y t , and finally use the "eos" symbol as the end mark to calculate the multi-character cross entropy loss function to realize the image to character encoding and decoding process.

[0045] Experimental results show that the proposed method achieves very high recognition accuracy on the CROHME dataset.

[0046] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A hand-drawn sketch recognition method based on multi-level stroke sequence feature extraction is characterized by The steps include: Step 1: Get a hand-drawn sketch; Step 2: Perform data augmentation on the hand-drawn sketch; Step 3: Construct a pluggable character-aware spatial channel attention mechanism. For the extracted hand-drawn sketch features, adaptively sequence the attention feature map information in the channel and space dimensions based on any convolutional neural network to obtain a new feature map. Step 4: Use a recurrent neural network character decoder to decode the new image features into character vectors, calculate the multi-character loss, and generate the final prediction.

2. The hand-drawn sketch recognition method based on multi-level stroke sequence feature extraction according to claim 1 is characterized in that: In the step 3, the extracted hand-drawn sketch features are input into the channel attention module, and a pooling operation based on the width and height of the feature map is performed. The pooled features are sent to the neural network, and channel attention features are generated based on the features output by the neural network and the hand-drawn sketch features. The channel attention features are input into the spatial attention module, and a channel-based pooling operation is performed to generate spatial attention features. Based on the channel attention features and the spatial attention features, the new feature map is generated.

3. The hand-drawn sketch recognition method based on multi-level stroke sequence feature extraction according to claim 2 is characterized in that: The channel attention module performs global maximum pooling and global average pooling based on width and height on the input hand-drawn sketch feature map F to obtain two 1×1×C feature maps F avg and F max , where C represents the number of channels; and a pooling parameter α is added to dynamically adjust the pooling weight. The formula after pooling is: F pool =α·F avg +(1-α)·F max Then, the obtained F pool They are sent to a two-layer neural network, the number of neurons in the first layer is C / r, r represents the reduction rate, and the number of neurons in the second layer is C; the output features are added and activated to generate feature W c ; Then, the channel self-attention feature F ′ =W c ⊙F Input spatial attention module.

4. The hand-drawn sketch recognition method based on multi-level stroke sequence feature extraction according to claim 2 is characterized in that: The spatial attention module performs channel-based global maximum pooling and global average pooling on the channel self-attention features to obtain two H×W×1 feature maps. After channel splicing operation, the two feature maps are reduced to H×W×1 through convolution operation, and spatial attention features are generated through activation operation.

5. The hand-drawn sketch recognition method based on multi-level stroke sequence feature extraction according to claim 2 is characterized in that: In step 3, the spatial attention feature M s and channel self-attention feature F ′ Do multiplication to get the final generated new feature F out =M s ⊙F ′ , ⊙ represents the main element multiplication.

6. The hand-drawn sketch recognition method based on multi-level stroke sequence feature extraction according to claim 1 is characterized in that: In step 4, feature decoding is performed using a gated recurrent unit (GRU) in a recurrent neural network. The GRU controls the update of the hidden state and the influence of the previous state, and the high-dimensional features are input into the GRU in a time series.

7. The hand-drawn sketch recognition method based on multi-level stroke sequence feature extraction according to claim 6 is characterized in that: The GRU receives the output y of the previous time step at each time point t-1 and the hidden state h t-1 , calculate the new hidden state h by updating the formula t : h t =z t ⊙h t-1 +(1-z t )⊙h t Then, the output is predicted based on the current hidden state through a fully connected layer: y t =softmax(Wyh t +b y ) In this process, character sequences are generated step by step until the end; the entire decoding process optimizes the model by calculating the multi-character cross entropy loss, which measures the gap between the predicted output and the true label.

8. The hand-drawn sketch recognition method based on multi-level stroke sequence feature extraction according to claim 1 is characterized in that: In step 1, the stroke trajectory coordinates are obtained through the hand-drawn sketch to construct a stroke data set corresponding to the hand-drawn sketch. The trajectory information of each stroke includes a starting point and an end point, and the stroke data of the unit sketch are unified into a matrix with a fixed dimension.

9. The hand-drawn sketch recognition method based on multi-level stroke sequence feature extraction according to claim 1, characterized in that: In the step 2, spatially deformable character data enhancement is used to expand the stroke data set, including: linear transformation of the hand-drawn sketch, stretching the hand-drawn sketch to simulate the breakage or thinning of the strokes, and simulating the thickening of the strokes or the diffusion of ink.

10. A hand-drawn sketch recognition method based on multi-level stroke sequence feature extraction according to claim 1, applied to hand-drawn formula recognition, characterized in that: In the step 1, the stroke data of each formula is unified into a matrix with a fixed dimension; in the step 3, a densely connected convolutional network is used.

Images