A fast brush font type recognition method integrating ViT and convolutional neural network

Abstract

The invention discloses a method for quickly identifying brush font types by integrating ViT and convolutional neural network, comprising: step 1: preprocessing brush font image data of multiple categories obtained from a calligraphy font library; step 2: adjusting the brightness, contrast saturation of the brush font image data; step 3: adjusting the brush font images in the data set to the same resolution format; step 4: sending the adjusted brush font image and annotation data to CNN to extract local feature information of the font; step 5: sending the adjusted brush font image and annotation data to ViT to extract structural feature information of the font; step 6: combining the local feature information and structural feature information of the font, and sending them to a classification model for training; the method achieves good recognition accuracy, greatly improves the speed and convenience of calligraphy font recognition, and provides technical guarantee for the development and application of intelligent devices.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method for quickly identifying brush font types by integrating vision transformer (ViT) and convolutional neural network (CNN). Background Art

[0002] Brush calligraphy is a traditional form of artistic expression of Chinese characters. There are five recognized styles in the calligraphy world, namely seal script, official script, regular script, cursive script and running script. Each font has its own unique writing style. Seal script is a pictographic and ideographic font, characterized by its flexible brushstrokes and lifelike appearance. Official script has distinct strokes, long horizontal strokes and broken vertical strokes, and pays attention to "silkworm head and goose tail" and "ups and downs". Regular script has a square shape and straight strokes, and pays attention to "plump and vigorous" and "strong structure". Cursive script is different from regular script in its regular brushwork, and it has the characteristics of "floating like clouds" and "correct like a startled dragon". Running script has both the regularity of regular script and the unrestrained flow of cursive script. It is a font that combines art and practicality. The styles of these five types of fonts are different, but they also have similarities.

[0003] At present, the recognition of calligraphy fonts mainly relies on the experience and judgment of professionals, but the manual method alone not only requires a lot of time and effort, but also has serious differences between different personnel. The font image contains a lot of glyph information, and it is difficult to grasp these tiny features with the naked eye. Beginners often have problems with writing irregularly because they do not understand the difference between fonts. Although ordinary deep learning methods can obtain more feature information of font images, they ignore the subtle features of font radicals and stroke order, which have an important impact on the judgment of brush font types. Summary of the invention

[0004] The purpose of the present invention is to propose a method for fast identification of brush font types based on fine-grained morphological information, which transforms the classification problem of brush fonts into an image classification problem in the field of artificial intelligence, and then obtains rich feature information in the font.

[0005] To achieve the above purpose, the present application proposes a method for quickly identifying brush font types by integrating ViT and convolutional neural network, including:

[0006] Step 1: Preprocess the brush font image data of various categories obtained from the calligraphy font library, and then classify and store them;

[0007] Step 2: Adjust the brightness, contrast and saturation of the classified and stored brush calligraphy image data, and randomly add noise to enhance and expand the data set;

[0008] Step 3: Adjust the calligraphy images in the dataset to the same resolution format, and divide them into a training set and a validation set according to a certain ratio;

[0009] Step 4: The adjusted brush font image and annotation data are sent to CNN to extract the local feature information of the font;

[0010] Step 5: The adjusted brush font image and annotation data are sent to ViT to extract the structural feature information of the font;

[0011] Step 6: combining the local feature information and the structural feature information of the font, sending the information to a classification model for training, and saving the classification model with the best effect;

[0012] Step 7: Load the classification model and pass the brush font image to be tested into the model to determine the category of the font.

[0013] Furthermore, in step 1, the multiple categories of brush font image data obtained from the calligraphy font library are preprocessed and then classified and stored, specifically including:

[0014] Step 1.1: Screen the brush font image data from the calligraphy font library and delete the data with low image quality;

[0015] Step 1.2: Collect and organize the image data of different types of calligraphy fonts and store them in folders corresponding to seal script, official script, regular script, cursive script and running script respectively;

[0016] Furthermore, in step 2, the brightness, contrast and saturation of the classified and stored brush font image data are adjusted, and noise is randomly added to enhance and expand the data set, specifically including:

[0017] Step 2.1: Manually clean the classified and stored brush calligraphy image data, and mark the wrong images by comparing them with the marked examples in the data set;

[0018] Step 2.2: For the calligraphy images in the data set, flip them with a probability of 50%. During the image flipping process, flip them horizontally and vertically with a probability of 50% respectively; for each calligraphy image, adjust the brightness, contrast and saturation with a probability of 100%. During the adjustment process, randomly change the brightness, contrast and saturation with a probability of 33.3% respectively;

[0019] Step 2.3: Add noise to the dataset with a probability of 30%. During the adding process, add Gaussian noise, pepper noise, and salt noise with a probability of 33.3% respectively; then retain the processed brush font image and the original brush font image, thereby achieving the enhancement and expansion of the dataset.

[0020] Furthermore, in step 3, the brush calligraphy images in the data set are adjusted to the same resolution format, and divided into a training set and a validation set according to a certain ratio, specifically including:

[0021] Step 3.1: Call the torchvision function library of the Python library to convert the resolution of the brush calligraphy image and unify it to 64×64;

[0022] Step 3.2: Randomly select some brush font image data and construct training set and validation set in a ratio of 8:2;

[0023] Step 3.3: Convert the brush font image data with unified resolution into Tensor format.

[0024] Furthermore, the brush font image and the annotated data adjusted in step 4 are sent to CNN to extract the local feature information of the font, including:

[0025] Step 4.1: Perform a convolution operation on the brush font image Tensor obtained in step 3. Considering that the appearance differences between different types of fonts are relatively subtle, a convolutional neural network with convolution kernels of different sizes is used to enhance the model's ability to extract features.

[0026] Step 4.2: Use batch normalization (BatchNorm) to standardize the input layer and hidden layer, use the visual activation function (FReLU) to accelerate convergence, pass it to the maximum pooling layer (MaxPool) for pooling operation, and obtain the local feature information of the font.

[0027] Furthermore, the brush font image and annotation data adjusted in step 5 are sent to ViT to extract the structural feature information of the font, including:

[0028] Step 5.1: Divide the brush calligraphy image Tensor obtained in step 3 into blocks, flatten it into sequences, and input it into the encoder part of the ViT model, which adopts a stacked 6-encoder module structure;

[0029] Step 5.2: Split the calligraphy image, re-encode the position, and use the learnable one-dimensional position embedding as the input of the position encoding;

[0030] Step 5.3: Divide the output after position encoding into 8 attention modules, obtain the value of each attention module, and concatenate them;

[0031] The attention module is spliced ​​using formula (1):

[0032] MultiHeadAttention(Q,K,V)=Concat(head1,...,head8) (1)

[0033] head i =Attention(QW i Q ,KW i K ,VW i V ) (2)

[0034] Among them, head i represents the attention module;

[0035] The value acquisition formula of the attention module is as follows:

[0036]

[0037] where d k represents the input dimension, Q, K, V represent the Query, Key and Value matrices respectively, T represents the transpose of the matrix, if Q*K T If the value of is too large, the partial derivative of softmax will approach 0. In order to solve this effect, it is multiplied by the scaling factor

[0038] Step 5.4: Make residual connections between the outputs of multiple attention modules and the outputs of position encoding. This structural design solves the problem of neural network degradation caused by too deep fully connected layers.

[0039] Step 5.5: The brush font image after residual connection is sent to the feedforward neural network, and finally layer normalization is performed to obtain the structural feature information of the font.

[0040] Furthermore, the specific formula for the residual connection and layer normalization integration is as follows:

[0041] Y=LayerNorm(X+MultiHeadAttention(X)) (4)

[0042] Among them, X is the feature vector after position encoding, and Y is the output after residual connection and layer normalization.

[0043] Furthermore, the feedforward neural network formula is as follows:

[0044] FeedForward(X)=max(0,XW1+b1)W2+b2 (5)

[0045] Where W i represents the weight of the fully connected layer, b irepresents the bias of the fully connected layer.

[0046] As a further step, in step 6, the local feature information and the structural feature information of the font are combined and sent to the classification model for training, and the classification model with the best effect is saved, which specifically includes:

[0047] Step 6.1: Flatten the vector output by CNN and concatenate it with the vector output by ViT. The formula for concatenating the two feature vectors is as follows:

[0048] FeatureUnion(X CNN ,X ViT )=Concat(Flatten(X CNN ),X ViT ) (6)

[0049] Step 6.2: The connected overall feature vector is passed into the fully connected neural network to train the classification model; in order to obtain the best model effect, the focal loss function (Focal Loss) is used to optimize the model and save the classification model with the best effect.

[0050] As a further step, in step 7, the classification model is loaded, and the brush font image to be tested is passed into the model to determine the category of the font, which specifically includes:

[0051] Load the classification model saved in step 6.2, input the brush font image to be classified into the model, determine the feature information of the font, and output the predicted font category.

[0052] Compared with the prior art, the above technical solution adopted by the present invention has the following advantages: the present invention first uses a mobile portable device to capture an image of a calligraphy font, then uses the image as input, uses the saved model to perform font recognition, and predicts the font corresponding to the image. It effectively corrects the problem of non-standard calligraphy, achieves good recognition accuracy, greatly improves the speed and convenience of calligraphy font recognition, and provides technical support for the development and application of smart devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is a flow chart of a method for rapid identification of brush font types that integrates ViT and convolutional neural network;

[0054] Figure 2 Model structure diagram for brush font classification;

[0055] Figure 3 This is a chart showing the classification of brush calligraphy fonts. Specific implementation methods

[0056] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application, that is, the embodiments described are only part of the embodiments of the present application, not all of the embodiments.

[0057] Example 1

[0058] Based on the stroke rules of different fonts and the current shortcomings of calligraphy tutoring tools, a method for quickly identifying brush font types by integrating ViT and convolutional neural network is provided. In this embodiment, PyCharm is used as the development platform, Python is used as the development language, and PyTorch is used as the deep learning framework. The above method of the present invention is adopted. Figure 1 As shown, font recognition and classification are performed. The specific process is as follows:

[0059] Step 1: Use a mobile phone, camera or other portable device to take a picture of the calligraphy font;

[0060] Step 2: Take the image obtained in step 1 as input, such as Figure 2 As shown, the classification model in this method is loaded to predict the classification results of the fonts in the image. The evaluation indicators used in the present invention include accuracy (ACC) and F1 value (F1 score);

[0061] The specific formula is as follows:

[0062]

[0063]

[0064]

[0065]

[0066] Among them, TP: True positive example, the number of positive classes correctly predicted as positive; FP: False positive example, the number of negative classes incorrectly predicted as positive; FN: False negative example, the number of positive classes incorrectly predicted as negative; TN: True negative example, the number of negative classes correctly predicted as negative.

[0067] Step 3: The recognition speed FPS (Frame per second, i.e. the number of images that can be processed per second) is used as the evaluation index of the model recognition speed. The formula is defined as follows:

[0068]

[0069] Among them, frameNum represents the total number of images that need to be inferred, and elapsedTime represents the total time (in seconds) of the inference process.

[0070] Step 4: Display the generated image structure output, such as Figure 3 and save it to a local file.

[0071] The foregoing description of specific exemplary embodiments of the present invention is for the purpose of illustration and demonstration. These descriptions are not intended to limit the present invention to the precise form disclosed, and it is clear that many changes and variations can be made based on the above teachings. The purpose of selecting and describing the exemplary embodiments is to explain the specific principles of the present invention and its practical application, so that those skilled in the art can realize and utilize various different exemplary embodiments of the present invention and various different selections and changes. The scope of the present invention is intended to be limited by the claims and their equivalents.

Claims

1. A method for fast identification of brush font types by integrating ViT and convolutional neural network, characterized in that: include: Step 1: Preprocess the brush font image data of various categories obtained from the calligraphy font library, and then classify and store them; Step 2: Adjust the brightness, contrast and saturation of the classified and stored brush calligraphy image data, and randomly add noise to enhance and expand the data set; Step 3: Adjust the calligraphy images in the dataset to the same resolution format, and divide them into a training set and a validation set according to a certain ratio; Step 4: The adjusted brush font image and annotation data are sent to CNN to extract the local feature information of the font; Step 5: The adjusted brush font image and annotation data are sent to ViT to extract the structural feature information of the font; Step 6: combining the local feature information and the structural feature information of the font, sending the information to a classification model for training, and saving the classification model with the best effect; Step 7: Load the classification model and pass the brush font image to be tested into the model to determine the category of the font.

2. According to claim 1, a method for quickly identifying brush font types by integrating ViT and convolutional neural network, characterized in that: In step 1, the image data of various types of brush fonts obtained from the calligraphy font library are preprocessed and then classified and stored, specifically including: Step 1.1: Screen the brush font image data from the calligraphy font library and delete the data with low image quality; Step 1.2: Collect and organize the image data of different types of calligraphy fonts, and store them in folders corresponding to seal script, official script, regular script, cursive script, and running script.

3. According to the method for rapid identification of brush font types by integrating ViT and convolutional neural network as described in claim 1, it is characterized in that: In step 2, the brightness, contrast and saturation of the classified and stored brush font image data are adjusted, and noise is randomly added to enhance and expand the data set, specifically including: Step 2.1: Manually clean the classified and stored brush calligraphy image data, and mark the wrong images by comparing them with the marked examples in the data set; Step 2.2: For the calligraphy images in the data set, flip them with a probability of 50%. During the image flipping process, flip them horizontally and vertically with a probability of 50% respectively; for each calligraphy image, adjust the brightness, contrast and saturation with a probability of 100%. During the adjustment process, randomly change the brightness, contrast and saturation with a probability of 33.3% respectively; Step 2.3: Add noise to the dataset with a probability of 30%. During the adding process, add Gaussian noise, pepper noise, and salt noise with a probability of 33.3% respectively; then retain the processed brush font image and the original brush font image, thereby achieving the enhancement and expansion of the dataset.

4. According to the method for fast identification of brush font types by integrating ViT and convolutional neural network as described in claim 1, it is characterized in that: In step 3, the brush font images in the data set are adjusted to the same resolution format and divided into a training set and a validation set according to a certain ratio, specifically including: Step 3.1: Call the torchvision function library of the Python library to convert the resolution of the brush calligraphy image and unify it to 64×64; Step 3.2: Randomly select some brush font image data and construct training set and validation set in a ratio of 8:2; Step 3.3: Convert the brush font image data with unified resolution into Tensor format.

5. According to claim 4, a method for quickly identifying brush font types by integrating ViT and convolutional neural network, characterized in that: The adjusted brush font image and annotation data in step 4 are sent to CNN to extract the local feature information of the font, including: Step 4.1: Perform a convolution operation on the brush calligraphy image Tensor obtained in step 3, that is, use a convolutional neural network with convolution kernels of different sizes to extract features; Step 4.2: Use batch normalization to standardize the input layer and hidden layer, use visual activation function to accelerate convergence, pass it to the maximum pooling layer for pooling operation, and obtain the local feature information of the font.

6. According to claim 4, a method for quickly identifying brush font types by integrating ViT and convolutional neural network, characterized in that: The adjusted brush font image and annotation data in step 5 are sent to ViT to extract the structural feature information of the font, including: Step 5.1: Divide the brush calligraphy image Tensor obtained in step 3 into blocks, flatten it into sequences, and input it into the encoder part of the ViT model, which adopts a stacked 6-encoder module structure; Step 5.2: Split the calligraphy image, re-encode the position, and use the learnable one-dimensional position embedding as the input of the position encoding; Step 5.3: Divide the output after position encoding into 8 attention modules, obtain the value of each attention module, and concatenate them; The attention module is spliced ​​using formula (1): MultiHeadAttention(Q,K,V)=Concat(head1,...,head8) (1) head i =Attention(QW i Q ,KW i K ,VW i V ) (2) Among them, head i represents the attention module; The value acquisition formula of the attention module is as follows: where d k represents the input dimension, Q, K, and V represent the Query, Key, and Value matrices respectively, and K T represents the transpose of the matrix K; Step 5.4: Make residual connections between the outputs of multiple attention modules and the outputs of position encoding; Step 5.5: The brush font image after residual connection is sent to the feedforward neural network, and finally layer normalization is performed to obtain the structural feature information of the font.

7. According to claim 6, a method for quickly identifying brush font types by integrating ViT and convolutional neural network, characterized in that: The specific formula for the residual connection and layer integration is as follows: Y=LayerNorm(X+MultiHeadAttention(X)) (4) Among them, X is the feature vector after position encoding, and Y is the output after residual connection and layer normalization.

8. According to claim 6, a method for quickly identifying brush font types by integrating ViT and convolutional neural network, characterized in that: The feedforward neural network formula is as follows: FeedForward(X)=max(0,XW1+b1)W2+b2 (5) Where W i represents the weight of the fully connected layer, b i represents the bias of the fully connected layer.

9. According to claim 1, a method for quickly identifying brush font types by integrating ViT and convolutional neural network, characterized in that: In step 6, the local feature information and structural feature information of the font are combined and sent to the classification model for training, and the classification model with the best effect is saved, which specifically includes: Step 6.1: Flatten the vector output by CNN and concatenate it with the vector output by ViT. The formula for concatenating the two feature vectors is as follows: FeatureUnion(X CNN ,X ViT )=Concat(Flatten(X CNN ),X ViT ) (6) Step 6.2: The connected overall feature vector is passed into the fully connected neural network to train the classification model; the focal loss function is used to optimize the model and the classification model with the best effect is saved.

10. According to claim 9, a method for quickly identifying brush font types by integrating ViT and convolutional neural network, characterized in that: In step 7, the classification model is loaded, and the brush font image to be tested is passed into the model to determine the category of the font, which specifically includes: Load the classification model saved in step 6.2, input the brush font image to be classified into the model, determine the feature information of the font, and output the predicted font category.

Images