A fine-grained Chinese calligraphy style recognition model construction method, system, device and storage medium

Abstract

The application discloses a fine-grained Chinese calligraphy style recognition model construction method, system, device and storage medium, and comprises the following steps: a data set containing works of multiple famous Chinese calligraphers is acquired and constructed; calligraphy pictures in the data set are preprocessed; a deep learning model is selected, the preprocessed data set is combined with a general image data set, and the deep learning model is retrained; the preprocessed data set is input into the model for CLS feature extraction, and then CLS features of all images are combined into a new CLS feature vector; principal component analysis dimension reduction is performed on the CLS feature vector; a Gaussian mixture model is used to fit the dimension-reduced CLS feature vector; and the log likelihood value of the preprocessed data set is calculated; and the model is not only limited to recognizing common font styles, but also can accurately recognize the style of a specific calligrapher.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image classification and recognition, and in particular to a fine-grained Chinese calligraphy style recognition model construction method, system, device and storage medium. BACKGROUND

[0002] In the technical field, the research on calligraphy recognition mainly focuses on how to extract and understand the handwriting, structure and style of calligraphy from images. This task is extremely challenging due to the diversity and complexity of calligraphy. Traditional machine learning methods rely on hand-designed features such as edges, corners and textures, or use classic machine learning algorithms such as support vector machines (SVM), random forests, etc. for classification, thus having the following limitations: 1) difficult to distinguish between works of different calligraphers under the same style or works of the same calligrapher at different periods; 2) insufficient adaptability to factors such as the preservation state of calligraphy works, paper aging and ink fading; 3) some deep learning models have high accuracy but require a large amount of computing resources, which may not be efficient in real-time recognition systems. SUMMARY

[0003] In view of the above defects or improvement needs of the prior art, the purpose of the present application is to provide a fine-grained Chinese calligraphy style recognition model construction method, system, device and storage medium. The method combines advanced deep learning technology and Gaussian mixture model, specifically uses ViT8B model based on Vision Transformer architecture to extract image features, and uses PCA dimensionality reduction and whitening processing to optimize data representation, enhancing the generalization ability and processing efficiency of the model, not only limited to common font styles, but also able to accurately identify the style of a specific calligrapher.

[0004] In a first aspect, the present application provides a fine-grained Chinese calligraphy style recognition model construction method, characterized in that it comprises:

[0005] 1. A fine-grained Chinese calligraphy style recognition model construction method, characterized in that it comprises:

[0006] S100, acquiring and constructing a data set containing works of multiple famous Chinese calligraphers;

[0007] S200, preprocessing the calligraphy pictures in the data set to obtain a preprocessed data set;

[0008] S300, selecting a deep learning model based on Vision Transformer architecture, combining the preprocessed data set with a general image data set, retraining the deep learning model, and obtaining a DINO-ViT model after training;

[0009] S400, input the preprocessed data set into the DINO-ViT model for CLS feature extraction, and then combine all the CLS features of the images into a new CLS feature vector;

[0010] S500, performing principal component analysis dimension reduction on the CLS feature vector to obtain a reduced CLS feature vector;

[0011] S600, fitting the reduced CLS feature vector using a Gaussian mixture model, and obtaining a mathematical model describing the Gaussian distribution of the CLS feature after fitting;

[0012] S700, calculating the log-likelihood value of the preprocessed data set using the mathematical model, storing the data, and completing the model construction.

[0013] Further, the preprocessing in step S300 includes size adjustment, normalization processing, and semantic segmentation processing to remove background interference.

[0014] Further, the principal component analysis dimension reduction in step S500 includes mapping high-dimensional features to a 64-dimensional space and performing whitening processing to remove the correlation between features and enhance the expression ability of the data.

[0015] Further, in the fitting process of step S600, the number of Gaussian distribution mixture components is defined as 16, and the maximum number of fitting iterations is set to 1000.

[0016] Further, step S600 further includes initializing the Gaussian distribution parameters in a random manner, including mean, standard deviation, and covariance matrix.

[0017] Further, step S600 further includes defining the covariance type as a full covariance matrix to capture the complex correlation between data features.

[0018] Further, the method further includes visualizing the log-likelihood distribution of different data sets through kernel density estimation and displaying on the log-likelihood histogram.

[0019] In a second aspect, the present application provides a fine-grained Chinese calligraphy style recognition device, characterized in that it comprises:

[0020] The first module is used for receiving a user inputted picture to be identified and performing preprocessing;

[0021] The second module is used for extracting a CLS feature vector in the picture to be identified and performing principal component analysis dimension reduction;

[0022] The third module is used for fitting the feature after principal component analysis dimension reduction to establish a mathematical model;

[0023] The fourth module is used for calculating a log-likelihood value of the picture to be recognized, performing comparison and judgment, and outputting a recognition result.

[0024] In a third aspect, the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any step of the fine-grained Chinese calligraphy style recognition model construction method when executing the computer program.

[0025] In a fourth aspect, the present application provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executable on a processor to implement any step of the fine-grained Chinese calligraphy style recognition model construction method.

[0026] The present application has the following advantages:

[0027] 1. The present application provides an innovative fine-grained Chinese calligraphy style recognition method, which combines advanced deep learning technology and Gaussian mixture model, specifically uses a ViT8B model of Vision Transformer architecture to extract image features, and uses PCA dimension reduction and whitening processing to optimize data representation, thereby enhancing the generalization ability and processing efficiency of the model, not only limiting to common font styles, but also accurately identifying the specific style of a calligrapher.

[0028] 2. The present application uses random initialization of GMM parameters and application of full covariance matrix to further improve the accuracy of model fitting and the ability to capture complex data distribution.

[0029] 3. The present application provides an intuitive visualization tool, which enables users to understand data distribution and model fitting through kernel density estimation graphics, thereby effectively identifying abnormal data.

[0030] Additional aspects and advantages of the present application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0031] The above and / or additional aspects and advantages of the present application will become apparent and be readily appreciated from the following description, taken in conjunction with the accompanying drawings, in which:

[0032] Figure 1 is a fine-grained Chinese calligraphy style recognition model construction method flowchart in an embodiment of the present application;

[0033] Figure 2 is a DINO-ViT model training schematic diagram in the embodiment of the present application;

[0034] Figure 3 is a log-likelihood visualization plot of the calligraphy dataset in embodiments of the present application. DETAILED DESCRIPTION

[0035] The application will be further described below in conjunction with the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the application, but not to limit the application. In addition, it should be noted that, for the convenience of description, only the parts related to the application are shown in the drawings, but not all the structures.

[0036] Those skilled in the art can understand that, unless specifically stated, the singular forms "a", "an" and "the" used herein also include the plural forms. It should be further understood that the phrase "comprising" used in the specification of the present application means that the features, integers, steps, operations, elements and / or components exist, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0037] Those skilled in the art can understand that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as that generally understood by those skilled in the art to which the present application belongs. It should also be understood that terms such as those defined in a general dictionary should be understood as having meanings consistent with those in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as such in the embodiments of the present application.

[0038] The present application provides a fine-grained Chinese calligraphy style recognition model construction method, system, device and storage medium, which combines advanced deep learning technology and Gaussian mixture model, specifically uses ViT8B model of VisionTransformer architecture to extract image features, and uses PCA dimension reduction and whitening processing to optimize data representation, enhance the generalization ability and processing efficiency of the model, not only limited to common font style, but also can accurately identify the specific style of a calligrapher.

[0039] Embodiment 1

[0040] As shown in Figure 1 The present application provides a fine-grained Chinese calligraphy style recognition model construction method, which comprises:

[0041] S100, acquiring and constructing a dataset containing works of multiple famous Chinese calligraphers;

[0042] S200, preprocessing the calligraphy pictures in the dataset to obtain a preprocessed dataset;

[0043] S300, select a deep learning model based on a Vision Transformer architecture, combine the preprocessed data set with a general image data set, retrain the deep learning model, and obtain a DINO-ViT model after training is completed;

[0044] S400, input the preprocessed data set into the DINO-ViT model for CLS feature extraction, and then combine all the CLS features of the images into a new CLS feature vector;

[0045] S500, perform principal component analysis dimension reduction on the CLS feature vector to obtain a reduced CLS feature vector;

[0046] S600, fit the reduced CLS feature vector using a Gaussian mixture model, and obtain a mathematical model that can describe the Gaussian distribution of the CLS feature after fitting;

[0047] S700, calculate the log-likelihood value of the preprocessed data set using the mathematical model, store the data, and complete the model construction;

[0048] After the model construction is completed, the data set is replaced with a picture to be identified, and preprocessed, steps S400-S700 are repeated, the log-likelihood value of the picture to be detected and the data set is compared, and the identification result is output.

[0049] Wherein, step S100 specifically comprises:

[0050] S101, construct a Chinese calligraphy style data set, this embodiment uses an open source data set for a Kaggle competition, which contains the works of 20 famous Chinese calligraphers, and aims to show the calligraphy styles of different calligraphers through these works.

[0051] Step S200 specifically comprises:

[0052] S201, size adjustment, because the DINO-ViT model is trained at 224*224 size, the picture to be detected needs to be resized to 224 when inputting data, so that the best performance can be obtained.

[0053] S202, normalization processing, because the DINO-ViT model training data uses ImageNet, in the image normalization process, the mean (0.485, 0.456, 0.406) and standard deviation (0.229, 0.224, 0.225) of the ImageNet data set are also used to normalize the image to achieve the best effect.

[0054] S203, semantic segmentation processing, if the paper of the calligraphy picture in the data set has very obvious paper texture, whether to perform matting can be set manually, the calligraphy part is obtained by semantic segmentation, and then superimposed on a pure white background picture to obtain a calligraphy picture without paper background interference, here semantic segmentation, an open source segmentation algorithm UNet is used, the pictures in the calligraphy data set are pixel-level labeled, and then a UNet segmentation model is trained.

[0055] Step S300 specifically includes:

[0056] S301, select DINO-ViTB8 model, which achieves 80.1% Top-1 accuracy on ImageNet classification task.

[0057] S302, the model is proposed and open sourced by Facebook AI Research (FAIR), and the open source model is trained based on ImageNet data set, which has high accuracy for general classification task, but since ImageNet does not include some calligraphy data, we need to add calligraphy data set classification data to ImageNet, and then train it in the general training way.

[0058] In particular, this model is designed to solve the general image recognition and classification problem, and the scheme here only uses the CLS feature extraction capability of the model. CLS feature is a special feature vector in ViT model, which is used to represent the global information of the whole image.

[0059] Step S400 specifically includes:

[0060] S401, input the image of all Chinese characters in a calligraphy work into the model one by one for feature extraction, then combine all the feature vectors of Chinese characters into a new feature vector, which can represent the style features of the whole calligraphy work; according to the method, the preprocessed Chinese calligraphy style data set is input into the model to extract feature vectors.

[0061] Step S500 specifically includes:

[0062] S501, since the CLS feature directly obtained by the model has relatively high dimension and may have some noise interference, this link reduces the dimension of the extracted feature, and the method used here is principal component analysis (PCA), which specifies to reduce the dimension of CLS feature to 64;

[0063] S502, the whitening processing is performed on the feature vector after dimension reduction to ensure that the data has unit variance and the characteristics are not related.

[0064] Step S600 specifically includes:

[0065] S601, receiving a dataset that has been processed by Principal Component Analysis (PCA) dimensionality reduction;

[0066] S602, defining the number of mixture components k of Gaussian distributions in the GMM as 16;

[0067] S603, setting the maximum number of iterations for GMM fitting to 1000;

[0068] S604, initializing the parameters of each Gaussian component in a random manner, including mean, standard deviation, and covariance matrix;

[0069] S605, defining the type of covariance matrix as full covariance matrix to capture complex correlations between data features;

[0070] S606, performing the GMM fitting process until the maximum number of iterations is reached or the model converges, obtaining a mathematical model that describes the data features.

[0071] wherein the Gaussian Mixture Model is defined by the following probability density function:

[0072]

[0073] wherein k represents the number of mixture components; πk is the weight of the kth Gaussian component; N(x | μk, ∑k) is a multivariate Gaussian distribution with mean μk and covariance ∑k as parameters; x represents an observed data point.

[0074] Further, the probability density function of each Gaussian distribution is:

[0075]

[0076] wherein D represents the dimension of the data, and |∑k| represents the determinant of the covariance matrix ∑k.

[0077] Step S700 specifically includes:

[0078] S701, using the log-likelihood value in the calligraphy dataset input by the fitted GMM model.

[0079] S702, according to the log-likelihood value, identifying and determining the style of the calligraphy work, and the identification process considers the high and low of the log-likelihood value, and the matching degree with the known calligrapher style model.

[0080] Further, the present embodiment also provides a result visualization and display, including:

[0081] As Figure 3As shown, the log-likelihood distributions of different calligrapher's works are visualized using Kernel Density Estimation (KDE) on a log-likelihood histogram.

[0082] In the graph, the X-axis represents the numerical value of log-likelihood, reflecting the fitting degree of data points in the model;

[0083] In the graph, the Y-axis represents the probability density at the corresponding log-likelihood value, reflecting the number of data points with that log-likelihood value.

[0084] Each curve represents the log-likelihood density estimation of different data sets, visualizing the distribution characteristics of the data sets.

[0085] For a normal distribution, if the data point x satisfies the following condition: μ-3σ ≤ x ≤ μ+3σ μ-3σ ≤ x ≤ μ+3σ then x is considered within the distribution, and if x exceeds this range, i.e. x < μ-3σ or x > μ+3σ x < μ-3σ or x > μ+3σ then x is considered outside the distribution.

[0086] Further, by comparing the curves of different data sets, we can evaluate their fitting in the GMM model. For example, if the curve of a certain abnormal data set is wider in the low log-likelihood value region, it may indicate that these data points do not belong to the data distribution used in model training, or they have significant differences with the distribution assumed by the model. Such visualization results can help us identify and understand the anomalies or unusual patterns in the data set. As Figure 3 As shown, the data distribution curves in the abnormal data set are all in the low log-likelihood value interval.

[0087] Further, the embodiment also provides a user interaction and feedback system, including:

[0088] Develop a graphical user interface (GUI) that allows users to upload calligraphy images, select different recognition options, and display recognition results.

[0089] The interface provides a feedback mechanism, allowing users to evaluate the recognition results, and the system learns and optimizes based on feedback.

[0090] Further, the embodiment also provides a system implementation and integration step, including:

[0091] Implement a backend processing system that integrates image preprocessing, model training, feature extraction, GMM modeling, recognition, and visualization modules.

[0092] Build a database to store calligrapher's works data, model parameters, user-uploaded images, and recognition results, supporting data management and analysis.

[0093] Further, the embodiment also provides an exception handling and system optimization mechanism, including:

[0094] An exception detection mechanism is designed to prompt the input image that does not meet the requirements and provide a manual correction option.

[0095] According to the recognition result and user feedback, the model parameters and algorithm are continuously optimized to improve the recognition accuracy and user experience of the system.

[0096] In some optional embodiments, the present application can verify the recognition ability of the model for calligraphy works under different preservation states, paper aging, ink fading, etc., to ensure that the model has good generalization.

[0097] In some optional embodiments, the present application can further consider the expansibility of the model, design a mechanism for model updating and new calligrapher style addition, to adapt to possible future data set expansion.

[0098] Further, the embodiment provides a fine-grained Chinese calligraphy style recognition device, including:

[0099] The first module is used for receiving user input of a to-be-recognized picture and performing preprocessing;

[0100] The second module is used for extracting a CLS feature vector in the to-be-recognized picture and performing principal component analysis dimension reduction;

[0101] The third module is used for fitting and establishing a mathematical model for the feature after principal component analysis dimension reduction;

[0102] The fourth module is used for calculating a log-likelihood value of the to-be-recognized picture, performing comparison and judgment, and outputting a recognition result.

[0103] Further, the embodiment also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor implements any step of the fine-grained Chinese calligraphy style recognition model construction method when executing the computer program.

[0104] Further, the embodiment also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement any step of the fine-grained Chinese calligraphy style recognition model construction method.

[0105] It should be understood that although the steps in the flowcharts of the drawings are shown in a sequential order following the arrows, the steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated otherwise herein, the execution of the steps is not strictly limited to the order indicated by the arrows, and can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the drawings can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of which is not necessarily sequential, but can be round-robin or alternating with at least some of the other steps or sub-steps or stages of other steps.

[0106] The above only describes some embodiments of the present application, and it should be pointed out that for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which should also be considered as the protection scope of the present application.

Claims

1. A method for constructing a fine-grained Chinese calligraphy style recognition model, characterized in that: include: S100, obtain and construct a dataset containing the works of many famous Chinese calligraphers; S200, preprocessing the calligraphy images in the data set to obtain a preprocessed data set; S300: Select a deep learning model based on the Vision Transformer architecture, combine the preprocessed dataset with a general image dataset, retrain the deep learning model, and obtain a DINO-ViT model after the training is completed; The method for obtaining the DINO-ViT model comprises: The DINO-ViTB8 model was selected as the base model; The images in the dataset are normalized using the mean and standard deviation of the ImageNet dataset; Combine the calligraphy dataset classification data with the ImageNet dataset for training; S400, inputting the preprocessed data set into the DINO-ViT model to extract CLS features, and then combining the CLS features of all images into a new CLS feature vector; The method of combining into a new CLS feature vector comprises: inputting images of all Chinese characters in a calligraphy work into a model one by one for feature extraction, and then combining the feature vectors of all Chinese characters into a new feature vector, which can represent the style characteristics of the entire calligraphy work; S500, performing principal component analysis dimensionality reduction on the CLS feature vector to obtain a CLS feature vector after dimensionality reduction; The principal component analysis dimensionality reduction includes: mapping high-dimensional features to a 64-dimensional space and performing whitening processing to remove the correlation between features and enhance the expressiveness of the data; S600, using a Gaussian mixture model to fit the CLS feature vector after dimensionality reduction, and obtaining a mathematical model that can describe the Gaussian distribution of the CLS feature after fitting; S700, using the mathematical model to calculate the log-likelihood value of the preprocessed data set, storing the data, and completing the model construction; The pre-processing in step S300 also includes: resizing and semantic segmentation processing to remove background interference; The semantic segmentation processing is specifically as follows: when the paper of the calligraphy image in the data set has paper texture, the calligraphy handwriting part is obtained by semantic segmentation, and then superimposed on a pure white background image to obtain a calligraphy image without paper background interference.

2. The method for constructing a fine-grained Chinese calligraphy style recognition model according to claim 1, characterized in that: In the fitting process of step S600, the number of Gaussian distribution mixture components is defined as 16, and the maximum number of fitting iterations is set to 1000.

3. The method for constructing a fine-grained Chinese calligraphy style recognition model according to claim 2, characterized in that: Step S600 further includes: initializing Gaussian distribution parameters, including mean, standard deviation, and covariance matrix, in a random manner.

4. The method for constructing a fine-grained Chinese calligraphy style recognition model according to claim 3, characterized in that: Step S600 further includes: defining the covariance type as a full covariance matrix to capture the complex correlations between data features.

5. A method for constructing a fine-grained Chinese calligraphy style recognition model according to any one of claims 1 to 4, characterized in that: The step S700 further includes: visualizing the log-likelihood distribution of different data sets through kernel density estimation, and displaying the results on a log-likelihood graph, so as to identify and understand abnormal or unusual patterns in the data sets.

6. A fine-grained Chinese calligraphy style recognition system, characterized by: include The first module is used to obtain and construct a dataset containing the works of many famous Chinese calligraphers; Preprocessing the calligraphy images in the data set to obtain a preprocessed data set; Select a deep learning model based on the VisionTransformer architecture, combine the preprocessed dataset with a general image dataset, retrain the deep learning model, and obtain the DINO-ViT model after training. The method for obtaining the DINO-ViT model comprises: The DINO-ViTB8 model was selected as the base model; The images in the dataset are normalized using the mean and standard deviation of the ImageNet dataset; Combine the calligraphy dataset classification data with the ImageNet dataset for training; The second module is used to input the preprocessed dataset into the DINO-ViT model to extract CLS features, and then combine the CLS features of all images into a new CLS feature vector; The method of combining into a new CLS feature vector comprises: inputting images of all Chinese characters in a calligraphy work into a model one by one for feature extraction, and then combining the feature vectors of all Chinese characters into a new feature vector, which can represent the style characteristics of the entire calligraphy work; Performing principal component analysis dimensionality reduction on the CLS feature vector to obtain a CLS feature vector after dimensionality reduction; The principal component analysis dimensionality reduction includes: mapping high-dimensional features to a 64-dimensional space and performing whitening processing to remove the correlation between features and enhance the expressiveness of the data; The third module is used to fit the CLS feature vector after dimensionality reduction using a Gaussian mixture model, and obtain a mathematical model that can describe the Gaussian distribution of the CLS feature after fitting; The fourth module is used to calculate the log-likelihood value of the preprocessed data set using the mathematical model, store the data, and complete the model construction; The preprocessing in the first module also includes: size adjustment and semantic segmentation processing to remove background interference; The semantic segmentation processing is specifically as follows: when the paper of the calligraphy image in the data set has paper texture, the calligraphy handwriting part is obtained by semantic segmentation, and then superimposed on a pure white background image to obtain a calligraphy image without paper background interference.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method for constructing a fine-grained Chinese calligraphy style recognition model as described in any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the method for constructing a fine-grained Chinese calligraphy style recognition model according to any one of claims 1 to 5 is implemented.

Images