Fine-grained Chinese calligraphy style recognition model construction method, system and device and storage medium

By combining deep learning and Gaussian mixed model, using ViT8B model to extract calligraphy image features and perform PCA dimensionality reduction, the problem of difficulty in distinguishing the works of different calligraphers in the existing technology and insufficient adaptability to the preservation state is achieved. Accurate and efficient recognition of calligraphy styles.

CN119027963AActive Publication Date: 2024-11-26WUHAN KEJIAN CULTURE TECH CO LTD

Patent Information

Application Number
CN202410861015.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-28
Publication Date
2024-11-26
Estimated Expiration
2044-06-28

AI Technical Summary

Technical Problem

In calligraphy recognition, it is difficult to distinguish between works of different calligraphers under the same style or works of different periods of the same calligrapher, and it is not adaptable to factors such as preservation status and paper aging. The deep learning model calculates resources in real-time recognition system. It consumes a lot and is not efficient.

Method used

Combining deep learning technology and Gaussian hybrid model, image features are extracted using ViT8B model of Vision Transformer architecture, and through PCA dimensionality reduction and whitening processing, combined with random initialization of GMM parameters and complete covariance matrix, a fine-grained Chinese calligraphy style recognition model is constructed and the model is enhanced. generalization ability and processing efficiency.

Benefits of technology

It realizes accurate identification of the style of specific calligraphers, improves the accuracy of model fitting and captures complex data distribution, provides intuitive visualization tools, and enhances the system's recognition efficiency and accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027963A_ABST
    Figure CN119027963A_ABST
Patent Text Reader

Abstract

The invention discloses a fine-grained Chinese calligraphy style recognition model construction method, system and device and a storage medium. The method comprises the steps of obtaining and constructing a data set containing multiple Chinese famous calligrapher works; preprocessing the calligraphy pictures in the data set; selecting a deep learning model, combining the preprocessed data set with a general image data set, and retraining the deep learning model; inputting the preprocessed data set into the model for CLS feature extraction, and then combining all the CLS features of all the images into a new CLS feature vector; performing principal component analysis dimensionality reduction on the CLS feature vector; a Gaussian mixture model is used to fit the CLS feature vectors after dimension reduction; calculating a log-likelihood value of the preprocessed data set; the model is not only limited to the recognition of common font styles, but also can more accurately recognize which calligrapher's style is.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image classification and recognition methods, and in particular to a method, system, device and storage medium for constructing a fine-grained Chinese calligraphy style recognition model. Background Art

[0003] In the technical field, research on calligraphy recognition mainly focuses on how to extract and understand the characteristics of calligraphy such as handwriting, structure and style from images. This task becomes extremely challenging due to the diversity and complexity of calligraphy. Traditional machine learning methods rely on hand-designed features such as edges, corners and textures, or use classic machine learning algorithms such as support vector machines (SVM) and random forests for classification. Therefore, they have the following limitations: 1) It is difficult to distinguish between works of different calligraphers in the same style or works of the same calligrapher in different periods; 2) It is not adaptable enough to factors such as the preservation state of calligraphy works, paper aging, and ink fading; 3) Although some deep learning models have high accuracy, they require a lot of computing resources and may not be efficient enough in real-time recognition systems. Summary of the invention

[0004] In view of the above defects or improvement needs of the prior art, the purpose of the present invention is to provide a method, system, device and storage medium for constructing a fine-grained Chinese calligraphy style recognition model. The method combines advanced deep learning technology and Gaussian mixture model, specifically utilizes the ViT8B model of the Vision Transformer architecture to extract image features, and uses PCA dimensionality reduction and whitening processing to optimize data representation, thereby enhancing the generalization ability and processing efficiency of the model. It is not only limited to common font styles, but can also accurately identify the style of a specific calligrapher.

[0005] In a first aspect, the present invention provides a method for constructing a fine-grained Chinese calligraphy style recognition model, characterized by comprising:

[0006] 1. A method for constructing a fine-grained Chinese calligraphy style recognition model, characterized by comprising:

[0007] S100, obtain and construct a dataset containing the works of many famous Chinese calligraphers;

[0008] S200, preprocessing the calligraphy images in the data set to obtain a preprocessed data set;

[0009] S300, selecting a deep learning model based on the Vision Transformer architecture, combining the preprocessed data set with a general image data set, retraining the deep learning model, and obtaining a DINO-ViT model after the training is completed;

[0010] S400, inputting the preprocessed data set into the DINO-ViT model to extract CLS features, and then combining the CLS features of all images into a new CLS feature vector;

[0011] S500, performing principal component analysis dimensionality reduction on the CLS feature vector to obtain a CLS feature vector after dimensionality reduction;

[0012] S600, using a Gaussian mixture model to fit the CLS feature vector after dimensionality reduction, and after fitting, a mathematical model that can describe the Gaussian distribution of CLS features is obtained;

[0013] S700, using the mathematical model to calculate the log-likelihood value of the preprocessed data set, storing the data, and completing model construction.

[0014] Furthermore, the preprocessing in step S300 includes: size adjustment, normalization processing, and semantic segmentation processing to remove background interference.

[0015] Furthermore, the principal component analysis dimensionality reduction in step S500 includes: mapping high-dimensional features to a 64-dimensional space and performing whitening processing to remove correlations between features and enhance the expressiveness of data.

[0016] Furthermore, in the fitting process of step S600, the number of Gaussian distribution mixture components is defined as 16, and the maximum number of fitting iterations is set to 1000.

[0017] Furthermore, step S600 also includes: initializing Gaussian distribution parameters in a random manner, including mean, standard deviation and covariance matrix.

[0018] Furthermore, step S600 also includes: defining the covariance type as a complete covariance matrix to capture the complex correlations between data features.

[0019] Furthermore, the method also includes: visualizing the log-likelihood distribution of different data sets through kernel density estimation and displaying it on a log-likelihood histogram.

[0020] In a second aspect, the present invention provides a fine-grained Chinese calligraphy style recognition device, characterized in that it includes:

[0021] The first module is used to receive the image to be identified input by the user and perform preprocessing;

[0022] The second module is used to extract the CLS feature vector in the image to be identified and perform principal component analysis and dimensionality reduction;

[0023] The third module: used to fit the features after the principal component analysis and dimension reduction to establish a mathematical model;

[0024] The fourth module is used to calculate the log-likelihood value of the image to be identified, make comparative judgments, and output the identification results.

[0025] In a third aspect, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any step in the method for constructing a fine-grained Chinese calligraphy style recognition model when executing the computer program.

[0026] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the computer program implements any step in the method for constructing a fine-grained Chinese calligraphy style recognition model as described above.

[0027] Beneficial effects of the present invention:

[0028] 1. The present invention provides an innovative fine-grained Chinese calligraphy style recognition method, which combines advanced deep learning technology and Gaussian mixture model, specifically uses the ViT8B model of the Vision Transformer architecture to extract image features, and uses PCA dimensionality reduction and whitening processing to optimize data representation, thereby enhancing the generalization ability and processing efficiency of the model. It is not only limited to common font styles, but can also accurately identify the style of a specific calligrapher.

[0029] 2. The present invention utilizes the application of randomly initialized GMM parameters and complete covariance matrix to further improve the accuracy of model fitting and the ability to capture complex data distribution.

[0030] 3. The present invention provides an intuitive visualization tool that enables users to understand data distribution and model fitting through kernel density estimation graphics, thereby effectively identifying abnormal data.

[0031] Additional aspects and advantages of the present application will be partially given in the following description, which will become apparent from the following description, or will be understood through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0033] Figure 1 is a flow chart of a method for constructing a fine-grained Chinese calligraphy style recognition model in an embodiment of the present invention;

[0034] Figure 2 is a schematic diagram of DINO-ViT model training in an embodiment of the present invention;

[0035] Figure 3 is a log-likelihood visualization curve graph of the calligraphy data set in an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0037] Those skilled in the art will appreciate that, unless otherwise stated, the singular forms "a", "an", "said" and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0038] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as the general understanding of those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and will not be interpreted with an idealized or overly formal meaning unless specifically defined as in the embodiments of this application.

[0039] The present invention provides a method, system, device and storage medium for constructing a fine-grained Chinese calligraphy style recognition model. The method combines advanced deep learning technology and a Gaussian mixture model, specifically utilizes the ViT8B model of the VisionTransformer architecture to extract image features, and uses PCA dimensionality reduction and whitening processing to optimize data representation, thereby enhancing the generalization ability and processing efficiency of the model. The method is not only limited to common font styles, but can also accurately identify the style of a specific calligrapher.

[0040] Example 1

[0041] like Figure 1 As shown, the embodiment of the present invention provides a method for constructing a fine-grained Chinese calligraphy style recognition model, comprising:

[0042] S100, obtain and construct a dataset containing the works of many famous Chinese calligraphers;

[0043] S200, preprocessing the calligraphy images in the data set to obtain a preprocessed data set;

[0044] S300, selecting a deep learning model based on the Vision Transformer architecture, combining the preprocessed data set with a general image data set, retraining the deep learning model, and obtaining a DINO-ViT model after the training is completed;

[0045] S400, inputting the preprocessed data set into the DINO-ViT model to extract CLS features, and then combining the CLS features of all images into a new CLS feature vector;

[0046] S500, performing principal component analysis dimensionality reduction on the CLS feature vector to obtain a CLS feature vector after dimensionality reduction;

[0047] S600, using a Gaussian mixture model to fit the CLS feature vector after dimensionality reduction, and after fitting, a mathematical model that can describe the Gaussian distribution of CLS features is obtained;

[0048] S700, using the mathematical model to calculate the log-likelihood value of the preprocessed data set, storing the data, and completing model construction;

[0049] After the model is built, the data set is replaced with the image to be identified, and preprocessing is performed. Steps S400-S700 are repeated to compare the log-likelihood value of the image to be detected with that in the data set, and the identification result can be output.

[0050] Wherein, step S100 specifically includes:

[0051] S101. Construct a Chinese calligraphy style dataset. This embodiment uses an open source dataset for the Kaggle competition, which contains works by 20 famous Chinese calligraphers. The purpose is to showcase the calligraphy styles of different calligraphers through these works.

[0052] Step S200 specifically includes:

[0053] S201, resize. Because the DINO-ViT model is trained on a size of 224*224, the image to be tested also needs to be resized to 224 when inputting data, so that the best performance can be achieved.

[0054] S202, perform normalization processing. Because ImageNet is used in the training data of the DINO-ViT model, in the process of image normalization, it is also necessary to use the mean (0.485, 0.456, 0.406) and standard deviation (0.229, 0.224, 0.225) of the ImageNet dataset to normalize the image to achieve the best effect.

[0055] S203, semantic segmentation processing. If the paper of the calligraphy image in the data set has very obvious paper texture, you can manually set whether to perform cutout, obtain the calligraphy part through semantic segmentation, and then superimpose it on a pure white background image to obtain a calligraphy image without paper background interference. Here, semantic segmentation uses the open source segmentation algorithm UNet to annotate the images in the calligraphy data set at the pixel level, and then train to obtain a UNet segmentation model.

[0056] Step S300 specifically includes:

[0057] S301. Select the DINO-ViTB8 model, which achieves 80.1% Top-1 accuracy on the ImageNet classification task.

[0058] S302, this model was proposed and open-sourced by Facebook AI Research (FAIR). The open-source model is trained based on the ImageNet dataset and has high accuracy for general classification tasks. However, ImageNet does not include some data about calligraphy. Therefore, we need to add the classification data of the calligraphy dataset to ImageNet and then conduct training using the general training method.

[0059] In particular, this model was originally designed to solve general image recognition and classification problems. Our solution here only uses the CLS feature extraction capability of the model. The CLS feature is a special feature vector in the ViT model that is used to represent the global information of the entire image.

[0060] Step S400 specifically includes:

[0061] S401. Input the images of all Chinese characters in a calligraphy work into the model one by one for feature extraction, and then combine the feature vectors of all Chinese characters into a new feature vector. This feature vector can represent the style characteristics of the entire calligraphy work; according to this method, input the preprocessed Chinese calligraphy style data set into the model to extract feature vectors.

[0062] Step S500 specifically includes:

[0063] S501. Since the CLS feature dimension directly obtained by the model is relatively high and there may be some noise interference, this step reduces the dimension of the features extracted above. The method used here is principal component analysis (PCA), which specifies that the CLS feature dimension is reduced to 64 dimensions.

[0064] S502: Perform whitening processing on the feature vector after dimensionality reduction to ensure that the data has unit variance and the features are uncorrelated.

[0065] Step S600 specifically includes:

[0066] S601, receiving a data set that has been processed by principal component analysis (PCA) for dimensionality reduction;

[0067] S602, define the number k of mixed components of Gaussian distribution in GMM as 16;

[0068] S603, setting the maximum number of iterations of GMM fitting to 1000;

[0069] S604, randomly initializing the parameters of each Gaussian component, including the mean, standard deviation and covariance matrix;

[0070] S605, defining the type of the covariance matrix as a complete covariance matrix to capture the complex correlations between data features;

[0071] S606, executing the GMM fitting process until the maximum number of iterations is reached or the model converges, and obtaining a mathematical model that describes the data characteristics.

[0072] Among them, the Gaussian mixture model is defined by the following probability density function:

[0073]

[0074] Where k is the number of mixture components; πk is the weight of the kth Gaussian component; N(x|μk,∑k) is a multivariate Gaussian distribution with mean μk and covariance ∑k as parameters; and x represents an observed data point.

[0075] Furthermore, the probability density function of each Gaussian distribution is:

[0076]

[0077] Where D represents the dimension of the data, and |∑k| represents the determinant of the covariance matrix ∑k.

[0078] Step S700 specifically includes:

[0079] S701. Utilize the log-likelihood value of the calligraphy data set input by the fitted GMM model.

[0080] S702. Identify and determine the style of the calligraphy work based on the log-likelihood value. The identification process takes into account the log-likelihood value and the degree of matching with the known calligrapher's style model.

[0081] Furthermore, this embodiment also provides a result visualization and display, including:

[0082] like Figure 3As shown, the log-likelihood distribution of works of different calligraphers is visualized using kernel density estimation (KDE) and displayed on the log-likelihood histogram.

[0083] Among them, in the graph, the X-axis represents the value of log likelihood, which reflects the degree of fit of the data point in the model;

[0084] In the graph, the Y-axis represents the probability density at the corresponding log-likelihood value, reflecting the number of data points with this log-likelihood value;

[0085] Each curve represents the log-likelihood density estimate of a different data set to visualize the distribution characteristics of the data set.

[0086] Among them, for a normal distribution, if the data point x satisfies the following conditions: μ-3σ≤x≤μ+3σμ-3σ≤x≤μ+3σ, then x is considered to be within the distribution, and if x is outside this range, that is, x<μ-3σ or x>μ+3μσx<μ-3σ or x>μ+3σ, then x is considered to be outside the distribution.

[0087] Furthermore, by comparing the curves of different datasets, we can evaluate how they fit the GMM model. For example, if the curve of an abnormal dataset is wider in the low log-likelihood value area, this may indicate that these data points do not belong to the data distribution used when the model was trained, or they are significantly different from the distribution assumed by the model. Such visualization results can help us identify and understand anomalies or unusual patterns in the dataset. Figure 3 As shown, the data distribution curves in the abnormal data set are all in the low log-likelihood value range.

[0088] Furthermore, this embodiment also provides a user interaction and feedback system, including:

[0089] Develop a graphical user interface (GUI) that allows users to upload calligraphy images, select different recognition options, and display the recognition results.

[0090] The interface provides a feedback mechanism, users can evaluate the recognition results, and the system learns and optimizes based on the feedback.

[0091] Furthermore, this embodiment also provides a system implementation and integration step, including:

[0092] Implement the back-end processing system, integrating image preprocessing, model training, feature extraction, GMM modeling, recognition and visualization modules.

[0093] Build a database to store calligraphers' work data, model parameters, user-uploaded images and recognition results, and support data management and analysis.

[0094] Furthermore, this embodiment also provides an exception handling and system optimization mechanism, including:

[0095] Design an anomaly detection mechanism to prompt input images that do not meet the requirements and provide manual correction options.

[0096] Based on the recognition results and user feedback, the model parameters and algorithms are continuously optimized to improve the system's recognition accuracy and user experience.

[0097] In some optional embodiments, the present invention can verify the model's ability to recognize calligraphy works under different preservation conditions, paper aging, ink fading, etc., to ensure that the model has good generalization.

[0098] In some optional embodiments, the present invention may further consider the extensibility of the model and design a mechanism for updating the model and adding new calligrapher styles to accommodate possible future expansion of the dataset.

[0099] Furthermore, an embodiment of the present invention provides a fine-grained Chinese calligraphy style recognition device, comprising:

[0100] The first module is used to receive the image to be identified input by the user and perform preprocessing;

[0101] The second module is used to extract the CLS feature vector in the image to be identified and perform principal component analysis and dimensionality reduction;

[0102] The third module: used to fit the features after the principal component analysis and dimension reduction to establish a mathematical model;

[0103] The fourth module is used to calculate the log-likelihood value of the image to be identified, make comparative judgments, and output the identification results.

[0104] Furthermore, an embodiment of the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any step in the method for constructing a fine-grained Chinese calligraphy style recognition model when executing the computer program.

[0105] Furthermore, an embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, any step in the method for constructing a fine-grained Chinese calligraphy style recognition model as described above is implemented.

[0106] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0107] The above description is only a partial implementation method of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for constructing a fine-grained Chinese calligraphy style recognition model, characterized in that: include: S100, obtain and construct a dataset containing the works of many famous Chinese calligraphers; S200, preprocessing the calligraphy images in the data set to obtain a preprocessed data set; S300, selecting a deep learning model based on the Vision Transformer architecture, combining the preprocessed data set with a general image data set, retraining the deep learning model, and obtaining a DINO-ViT model after the training is completed; S400, inputting the preprocessed data set into the DINO-ViT model to extract CLS features, and then combining the CLS features of all images into a new CLS feature vector; S500, performing principal component analysis dimensionality reduction on the CLS feature vector to obtain a CLS feature vector after dimensionality reduction; S600, using a Gaussian mixture model to fit the CLS feature vector after dimensionality reduction, and after fitting, a mathematical model that can describe the Gaussian distribution of CLS features is obtained; S700, using the mathematical model to calculate the log-likelihood value of the preprocessed data set, storing the data, and completing model construction.

2. The method for constructing a fine-grained Chinese calligraphy style recognition model according to claim 1, characterized in that: The preprocessing in step S300 includes: resizing, normalization, and semantic segmentation to remove background interference.

3. The method for constructing a fine-grained Chinese calligraphy style recognition model according to claim 1, characterized in that: The principal component analysis dimensionality reduction in step S500 includes: mapping high-dimensional features to a 64-dimensional space and performing whitening processing to remove the correlation between features and enhance the expressiveness of the data.

4. The method for constructing a fine-grained Chinese calligraphy style recognition model according to claim 1, characterized in that: In the fitting process of step S600, the number of Gaussian distribution mixture components is defined as 16, and the maximum number of fitting iterations is set to 1000.

5. The method for constructing a fine-grained Chinese calligraphy style recognition model according to claim 4, characterized in that: Step S600 also includes: initializing Gaussian distribution parameters in a random manner, including mean, standard deviation and covariance matrix.

6. The method for constructing a fine-grained Chinese calligraphy style recognition model according to claim 5, characterized in that: Step S600 also includes: defining the covariance type as a complete covariance matrix to capture the complex correlations between data features.

7. A method for constructing a fine-grained Chinese calligraphy style recognition model according to any one of claims 1-6, characterized in that: The step S700 also includes: visualizing the log-likelihood distribution of different data sets through kernel density estimation and displaying them on a log-likelihood curve graph to identify and understand abnormal or unusual patterns in the data sets.

8. A fine-grained Chinese calligraphy style recognition system, characterized in that: include The first module is used to receive the image to be identified input by the user and perform preprocessing; The second module is used to extract the CLS feature vector in the image to be identified and perform principal component analysis and dimensionality reduction; The third module: used to fit the features after the principal component analysis and dimension reduction to establish a mathematical model; The fourth module is used to calculate the log-likelihood value of the image to be identified, make comparative judgments, and output the identification results.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for constructing a fine-grained Chinese calligraphy style recognition model as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the method for constructing a fine-grained Chinese calligraphy style recognition model as described in any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Method for performing retrieval feature principal component analysis by using classification layer weight

    CN113934881A

  • Image recognition model acquisition method, image recognition method, device and medium

    CN114283310A

  • ViT and convolutional neural network fused calligraphy body type rapid identification method

    CN114677687A

  • New energy scene processing method and device based on principal component Gaussian hybrid clustering

    CN116166955A

  • Ancient ceramic element detection method and system, electronic equipment and storage medium

    CN117272061A

Cited By

  • Crop identification method and system based on OpenPCS and deep learning

    CN121600414A

  • Crop recognition method and system based on open pcs and deep learning

    CN121600414B