Calligraphy evaluation method and system of variational auto-encoder based on category constraint

Abstract

The invention discloses a calligraphy evaluation method and system of a variational auto-encoder based on category constraint, and relates to the technical field of calligraphy art automatic evaluation, and the method comprises the steps: generating potential variable representation of a Chinese character image and a corresponding evaluation text through feature extraction and compression of the image and the text; the method comprises the following steps: constructing consistency constraint and semantic relevance of an image and a text by utilizing category offset and a bidirectional long-short-term memory network decoder, and reflecting different levels of Chinese character writing quality by adjusting a mean value of hidden variables; and the KL divergence between the posterior distribution and the prior distribution is utilized to constrain hidden variable generation evaluation. According to the method, a classification constraint mechanism is introduced on the basis of a classic variational auto-encoder model architecture, so that the accuracy and reliability of the model in Chinese character writing evaluation are remarkably improved, and the professionality and authority of an evaluation result are ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of automated calligraphy art evaluation, and in particular to a calligraphy evaluation method and system based on a category-constrained variational autoencoder. Background Art

[0002] In the current field of automated calligraphy evaluation, although a variety of technologies have been applied, including skeleton extraction, stroke analysis, aesthetic feature quantification, and deep learning frameworks using convolutional neural networks, each technology has its own unique advantages and limitations.

[0003] First, skeleton extraction technology is commonly used in the field of calligraphy copying. It helps analyze and compare calligraphy works by extracting the main structural skeleton of Chinese characters. This method can effectively calculate the similarity between calligraphy works and standard or excellent works, and thus be used for copying training and evaluation. However, skeleton extraction technology mainly focuses on the macro structure of calligraphy works, ignoring the complex interaction between stroke details and structure, making it difficult to fully evaluate the artistic value and technical accuracy of Chinese characters.

[0004] Secondly, stroke analysis and aesthetic feature quantification methods focus on evaluating calligraphy works according to a series of preset criteria. Stroke analysis techniques mainly focus on the shape, thickness and strength of each stroke, while aesthetic feature quantification attempts to convert the beauty of calligraphy works into quantifiable data. The main problem with these techniques is that they are highly dependent on image quality and manually set evaluation criteria, which not only makes the evaluation results very sensitive to the quality of the image, but also may lead to inconsistent evaluation results due to the subjectivity of the criteria. In practical applications, especially when the image quality is not ideal, the accuracy and reliability of these methods will be greatly reduced.

[0005] Finally, convolutional neural network (CNN) methods are commonly used for classification tasks in the automated evaluation of calligraphy art, such as distinguishing different calligraphy styles or writers. This technology is able to effectively recognize and classify complex visual patterns by learning from a large amount of calligraphy image data. However, although convolutional neural networks have excelled in the field of image recognition, they have encountered difficulties in directly evaluating the aesthetic and technical quality of calligraphy works. This is mainly because calligraphy evaluation is not just a simple classification problem, it requires a deep understanding of the artistry, technical details and overall coordination of the work, which are difficult to achieve with current CNN models.

[0006] In summary, although the existing technologies provide certain support for the automated evaluation of calligraphy art, their respective limitations still make it difficult to achieve ideal results in the field of Chinese character evaluation. Summary of the invention

[0007] In view of the above-mentioned problems, the present invention is proposed.

[0008] Therefore, the technical problem solved by the present invention is that the existing automated evaluation methods for calligraphy art have difficulties in comprehensively evaluating the artistic value and technical accuracy of Chinese characters, have high requirements for image quality, and have problems with how to have an in-depth understanding of the artistry, technical details and overall coordination of the work.

[0009] To solve the above technical problems, the present invention provides the following technical solutions: a calligraphy evaluation method based on a category-constrained variational autoencoder, comprising generating latent variable representations of Chinese character images and corresponding evaluation texts through feature extraction and compression of images and texts; using category offsets and bidirectional long short-term memory network decoders to construct consistency constraints and semantic associations between images and texts, and reflecting different levels of Chinese character writing quality by adjusting the mean of latent variables; and using the KL divergence between the posterior distribution and the prior distribution to constrain latent variables to generate evaluations.

[0010] As a preferred solution of the calligraphy evaluation method based on the category-constrained variational autoencoder of the present invention, wherein: the latent variable representation of the Chinese character image and the corresponding evaluation text is generated by extracting and compressing the features of the image and the text, including the image encoder receiving the Chinese character image as input, using the residual network as the backbone network, extracting features and compressing the dimension of the input image, and obtaining the compressed representation h of the image img

[0011] h img =ResNet50(img)

[0012] The image encoder uses two independent fully connected layers FC to transform h img Mapped to the mean μ and logarithmic standard deviation logσ of the latent variable z, expressed as:

[0013] μ img =FC μ (h img )

[0014] logσ img =FC σ (h img )

[0015] Among them, μ img and logσ img They respectively represent the mapping of the compressed image to the mean and logarithmic standard deviation of the latent variable z.

[0016] As a preferred solution of the calligraphy evaluation method of the variational autoencoder based on category constraints described in the present invention, wherein: the latent variable representation of the Chinese character image and the corresponding evaluation text generated by extracting and compressing the features of the image and text includes adjusting the latent variable using the pre-defined category information in the data set after the image encoder encodes the input image to obtain the latent variable representation, introducing prior knowledge, and when preparing the data set, all Chinese characters are divided into three levels of good, medium and poor according to the writing quality, and are respectively assigned three category labels of 0, 1 and 2. For each category, a different offset shift is set, and the latent variable encodings of different categories are shifted in different directions by subtracting the offset of the corresponding category from the mean vector of the latent variable, which is expressed as:

[0017] z img =(μ img +shift)+σ img ⊙∈,

[0018] As a preferred solution of the calligraphy evaluation method based on the category-constrained variational autoencoder of the present invention, wherein: the use of the category offset and the bidirectional long short-term memory network decoder to construct the consistency constraint and semantic relevance between the image and the text includes the text encoder receiving the evaluation text corresponding to the Chinese character image as input, using the LSTM network to extract features and semantically encode the input evaluation, and obtaining a compressed representation of the text h text , expressed as:

[0019] h text =LSTM(text)

[0020] The text encoder uses two independent fully connected layers to transform h text Mapped to the latent variable mean μ text and logarithmic standard deviation, at the mean parameter μ text The category shift that reflects the quality level of Chinese characters is incorporated into the image to achieve the consistency constraint of the text information, and the sampling result z is generated by reparameterization. text , expressed as:

[0021] μ text =FC μ (h text )

[0022] logσ text =FC σ (h text )

[0023] z text =(μ text +shift)+σ text ⊙∈,

[0024] As a preferred solution of the calligraphy evaluation method of the variational autoencoder based on category constraints described in the present invention, wherein: the use of category offset and bidirectional long short-term memory network decoder to construct consistency constraints and semantic relevance between image and text also includes using the same bidirectional long short-term memory network as a decoder to respectively generate the potential representation z of the image encoder and the text encoder img and z text Decode and get dec img and dec text ;

[0025] z img Input into the BiLSTM decoder, after multiple layers of nonlinear transformation, the image decoding result dec is obtained img , expressed as:

[0026] dec img =BiLSTM(z img )

[0027] z text Input into the same BiLSTM decoder to get the text decoding result dec text , expressed as:

[0028] dec text =BiLSTM(z text )

[0029] Using two independent fully connected layers, dec img and dec text Mapped to the word table space, and the generation probability of each word is obtained through the normalization function, expressed as:

[0030] p(x t |z img )=softmax(FC(dec img ))

[0031] p(x t |z text )=softmax(FC(dec text ))

[0032] where x t Represents the tth word generated.

[0033] As a preferred solution of the calligraphy evaluation method of the variational autoencoder based on category constraints described in the present invention, wherein: the different levels of Chinese character writing quality reflected by adjusting the mean of the latent variables includes constructing training losses including reconstruction loss, image potential representation loss and text potential representation loss.

[0034] The reconstruction loss measures the difference between the decoder generated results and the real text. img and dec text With real text input text The cross entropy loss between is expressed as:

[0035]

[0036] in, represents the cross entropy loss function, expressed as

[0037]

[0038] Among them, p(x) represents the true value, which means the real evaluation text in this project, and q(x) represents the predicted value;

[0039] The image latent representation loss is expressed as:

[0040]

[0041] Among them, q(z) represents the posterior distribution and p(z) represents the prior distribution.

[0042] As a preferred solution of the calligraphy evaluation method of the variational autoencoder based on category constraints described in the present invention, wherein: the KL divergence between the posterior distribution and the prior distribution is used to constrain the latent variable generation evaluation including the image potential representation loss for constraining the potential representation generated by the image encoder, introducing a prior distribution, and minimizing z img The KL divergence between the prior distribution is expressed as:

[0043] Llatent img =D KL (q(z img ∣x img )||p(z img ))

[0044] Among them, q(z img |x img ) represents the posterior distribution generated by the image encoder, p(z img ) represents the prior distribution generated by the image encoder, which ensures that the generated z img Conform to the preset prior distribution;

[0045] The text potential representation loss is used to constrain the potential representation z generated by the text encoder text , expressed as:

[0046] Llatent text =D KL (q(ztext ∣x text )||p(z text ))

[0047] Among them, q(z text |x text ) represents the posterior distribution generated by the text encoder, p(z text) Represents the prior distribution generated by the text encoder, minimizing the text potential representation loss to ensure the generated z text In line with the preset prior distribution,

[0048] The overall loss function of the model is the weighted sum of reconstruction loss, image potential representation loss, and text potential representation loss, expressed as:

[0049]

[0050] Among them, λ is the weight coefficient of the reconstruction loss term. By balancing the three loss functions, the model simultaneously learns the potential representation of images and texts to generate evaluation text.

[0051] Another object of the present invention is to provide a calligraphy evaluation system based on a category-constrained variational autoencoder, which can significantly improve the accuracy and reliability of the model in Chinese character writing evaluation through a classification constraint mechanism, ensuring the professionalism and authority of the evaluation results. This solves the problem of insufficient accuracy in the current classic variational autoencoder model architecture.

[0052] As a preferred solution of the calligraphy evaluation system of the variational autoencoder based on category constraints described in the present invention, it includes: a data generation module, a rating module, and an evaluation module; the data generation module is used to generate latent variable representations of Chinese character images and corresponding evaluation texts by extracting and compressing features of images and texts; the rating module is used to use category offsets and bidirectional long short-term memory network decoders to construct consistency constraints and semantic associations between images and texts, and to reflect different levels of Chinese character writing quality by adjusting the mean of latent variables; the evaluation module is used to use the KL divergence between the posterior distribution and the prior distribution to constrain latent variables to generate evaluations.

[0053] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of a calligraphy evaluation method based on a variational autoencoder with category constraints.

[0054] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a calligraphy evaluation method based on a variational autoencoder with category constraints.

[0055] Beneficial effects of the present invention: The calligraphy evaluation method of the variational autoencoder based on category constraints provided by the present invention endows the model with the ability of self-learning and feature extraction, enabling it to independently discover and construct internal connections from image and text data, and realize an end-to-end intelligent learning mechanism from input to output. On top of the classic variational autoencoder model architecture, a classification constraint mechanism is introduced, which significantly improves the accuracy and reliability of the model in the evaluation of Chinese character writing, and ensures the professionalism and authority of the evaluation results. The present invention achieves better results in terms of accuracy and reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0057] Figure 1 An overall flow chart of a calligraphy evaluation method based on a variational autoencoder with category constraints provided in the first embodiment of the present invention.

[0058] Figure 2 A model flow chart of a calligraphy evaluation method based on a variational autoencoder with category constraints provided in the first embodiment of the present invention.

[0059] Figure 3 A diagram showing the operating effect of a calligraphy evaluation method based on a variational autoencoder with category constraints provided in the second embodiment of the present invention.

[0060] Figure 4 An overall flow chart of a calligraphy evaluation system based on a category-constrained variational autoencoder provided in the third embodiment of the present invention. DETAILED DESCRIPTION

[0061] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0062] Example 1

[0063] Reference Figure 1-Figure 2 , as an embodiment of the present invention, provides a calligraphy evaluation method of a variational autoencoder based on category constraints, comprising:

[0064] S1: Generate latent variable representation of Chinese character images and corresponding evaluation texts through feature extraction and compression of images and texts.

[0065] Furthermore, by extracting and compressing the features of images and texts, the latent variable representation of Chinese character images and corresponding evaluation texts is generated, including an image encoder receiving Chinese character images as input, using a residual network as a backbone network, extracting features and compressing dimensions of the input image, and obtaining a compressed representation of the image h img

[0066] h img =ResNet50(img)

[0067] The image encoder uses two independent fully connected layers FC to transform h img Mapped to the mean μ and logarithmic standard deviation logσ of the latent variable z, expressed as:

[0068] μ img =FC μ (h img )

[0069] logσ img =FC σ (h img )

[0070] Among them, μ img and logσ img They respectively represent the mapping of the compressed image to the mean and logarithmic standard deviation of the latent variable z.

[0071] It should be noted that the latent variable representation of Chinese character images and corresponding evaluation texts is generated by extracting and compressing features of images and texts. After the image encoder encodes the input image to obtain the latent variable representation, the latent variable is adjusted using the pre-defined category information in the data set. Prior knowledge is introduced. When preparing the data set, all Chinese characters are divided into three levels of good, medium, and poor according to the writing quality, and are assigned three category labels of 0, 1, and 2 respectively. For each category, a different offset shift is set. By subtracting the offset of the corresponding category from the mean vector of the latent variable, the latent variable encoding of different categories is shifted in different directions, which is expressed as:

[0072] z img =(μ img +shift)+σ img ⊙∈,

[0073] The motivation for introducing the category offset shift is that the evaluation of the quality of Chinese character writing is essentially a continuously changing process, and there is no clear dividing line between good and bad. This feature is different from traditional tasks (such as cat and dog generation) - cats and dogs are two independent species categories, and they should be relatively separable in the feature space. However, if the Chinese character image is directly mapped to the latent space and its inherent continuity is ignored, the reconstructed samples may not accurately reflect the quality differences of the original input. Therefore, this operation can make the samples generated by the variational autoencoder model better conform to the predefined category characteristics and improve the controllability of the generated samples. The evaluation is roughly divided into the following aspects:

[0074] The comments on Chinese characters in this document mainly focus on the following aspects:

[0075] 1. Structural proportion: Evaluate whether the structure of the Chinese character is appropriate, whether the center of gravity is stable, and whether the font is symmetrical and beautiful.

[0076] 2. Stroke shape: Comment on whether the strokes are precise, beautiful in shape, and whether there is a good response between the strokes.

[0077] 3. Stroke fluency: Evaluate whether the strokes are vivid, smooth, and dynamic when writing.

[0078] 4. Changes in stroke weight: Indicates whether the strokes are light or heavy, that is, whether the strength changes when writing, so as to show the rhythm and rhythm of calligraphy.

[0079] 5. Artistic personality: Evaluate whether the Chinese characters have a unique artistic style and personality.

[0080] 6. Structural rationality: Evaluate whether the structural layout of the Chinese characters is reasonable, and whether there are problems such as looseness, crookedness or disproportion.

[0081] 7. Stroke angle and support: Pay special attention to whether the angles of horizontal and vertical strokes are appropriate and whether the inclined support is sufficient to ensure the stability of the font.

[0082] 8. Writing characteristics: Evaluate whether the writing can be done according to the characteristics of the character shape, whether the character shape is too uniform and lacks the awareness of contraction and expansion.

[0083] 9. Clarity of strokes: Evaluate whether the strokes are clear, light and thin, and whether they are sloppy.

[0084] 10. Font clarity and orderliness: Evaluate whether the overall font is clear, orderly, or messy and disordered.

[0085] S2: Use category offset and bidirectional long short-term memory network decoder to build consistency constraints and semantic relevance between images and texts, and adjust the mean of latent variables to reflect different levels of Chinese character writing quality.

[0086] Furthermore, the class offset and bidirectional long short-term memory network decoder are used to construct the consistency constraint and semantic relevance between the image and the text. The text encoder receives the evaluation text corresponding to the Chinese character image as input, and uses the LSTM network to extract features and semantically encode the input evaluation to obtain the compressed representation of the text h text , expressed as:

[0087] h text =LSTM(text)

[0088] The text encoder uses two independent fully connected layers to transform h text Mapped to the latent variable mean μ text and logarithmic standard deviation, at the mean parameter μ text The category shift that reflects the quality level of Chinese characters is incorporated into the image to achieve the consistency constraint of the text information, and the sampling result z is generated by reparameterization. text , expressed as:

[0089] μ text =FC μ (h text )

[0090] logσ text =FC σ (h text )

[0091] z text =(μ text +shift)+σ text ⊙∈,

[0092] It should be noted that using the category offset and the bidirectional long short-term memory network decoder to construct the consistency constraint and semantic association between the image and the text also includes using the same bidirectional long short-term memory network as a decoder to respectively generate the potential representation z of the image encoder and the text encoder. img and z text Decode and get dec img and dec text The purpose of this design is to enable the decoder to simultaneously learn the potential representations of images and texts and capture the relationship between them. Because the difficulty of decoding is generally greater than that of encoding, a more complex network is used as the decoder.

[0093] z img Input into the BiLSTM decoder, after multiple layers of nonlinear transformation, the image decoding result dec is obtained img , expressed as:

[0094] decimg =BiLSTM(z img )

[0095] z text Input into the same BiLSTM decoder to get the text decoding result dec text , expressed as:

[0096] dec text =BiLSTM(z text )

[0097] Using two independent fully connected layers, dec img and dec text Mapped to the word table space, and the generation probability of each word is obtained through the normalization function, expressed as:

[0098] p(x t |z img )=softmax(FC(dec img ))

[0099] p(x t |z text )=softmax(FC(dec text ))

[0100] where x t Represents the tth word generated. It should be noted that although the decoder decodes the potential representations of the image and text at the same time during the training process, in actual applications, we only need to use the image encoder to compress and encode the input Chinese character image, and then input the obtained input into the decoder to generate the corresponding evaluation text. This is because the decoder has learned the association between the image and text representations, allowing it to infer the corresponding evaluation based only on the image information.

[0101] S3: Use the KL divergence between the posterior distribution and the prior distribution to constrain the latent variable generation evaluation.

[0102] Furthermore, the training loss is constructed by adjusting the mean of the latent variables to reflect the different levels of Chinese character writing quality, including reconstruction loss, image latent representation loss and text latent representation loss.

[0103] The reconstruction loss measures the difference between the decoder generated results and the real text. img and dec text With real text input text The cross entropy loss between is expressed as:

[0104]

[0105] in, represents the cross entropy loss function, expressed as

[0106]

[0107] Among them, p(x) represents the true value, which means the real evaluation text in this project, and q(x) represents the predicted value.

[0108] By minimizing the reconstruction loss, the decoder can generate results that are as close as possible to the real text.

[0109] The image latent representation loss is expressed as:

[0110]

[0111] Among them, q(z) represents the posterior distribution and p(z) represents the prior distribution.

[0112] It should be noted that the KL divergence between the posterior distribution and the prior distribution is used to constrain the latent variable generation evaluation, including the image potential representation loss used to constrain the potential representation generated by the image encoder, introduce a prior distribution, and minimize z img The KL divergence between and the prior distribution is expressed as:

[0113] Llatent img =D KL (q(z img ∣x img )||p(z img ))

[0114] Among them, q(z img |x img ) represents the posterior distribution generated by the image encoder, p(z img ) represents the prior distribution generated by the image encoder, which ensures that the generated z img It conforms to the preset prior distribution.

[0115] The text potential representation loss is used to constrain the potential representation z generated by the text encoder text , expressed as:

[0116] Llatent text =D KL (q(z text ∣x text )||p(z text ))

[0117] Among them, q(z text |x text ) represents the posterior distribution generated by the text encoder, p(z text) represents the prior distribution generated by the text encoder, minimizing the text potential representation loss to ensure the generated z text In line with the preset prior distribution,

[0118] The overall loss function of the model is the weighted sum of reconstruction loss, image potential representation loss, and text potential representation loss, expressed as:

[0119]

[0120] Among them, λ is the weight coefficient of the reconstruction loss term. By balancing the three loss functions, the model simultaneously learns the potential representation of images and texts to generate evaluation text.

[0121] Example 2

[0122] Reference Figure 3 , which is an embodiment of the present invention, provides a calligraphy evaluation method based on a variational autoencoder with category constraints. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments.

[0123] First, in order to comprehensively evaluate the comprehensive performance of the invention model, four evaluation indicators were selected: BLEU-4, METEOR, ROUGE and CIDEr. BLEU-4 mainly measures the overlap of vocabulary and phrases between the generated text and the reference text, focusing on the accuracy of grammatical structure and sentence structure. METEOR accurately evaluates the semantic consistency and diversity of vocabulary usage of translation through detailed synonym and stem matching, as well as word order considerations. ROUGE focuses on the coverage of content, especially the ability to capture key information, to evaluate the comprehensiveness of content. CIDEr evaluates the quality and amount of information in text description by analyzing the specificity of vocabulary and the richness of expression. These four indicators together constitute a comprehensive evaluation system to ensure that the quality of model output is evaluated from multiple dimensions.

[0124] Table 1 Experimental data comparison table

[0125]

[0126]

[0127] In order to objectively evaluate the performance of the VAE-shift model, the present invention introduces a baseline model that does not use the shift operation, and uses it as a comparative reference. The corresponding evaluation indicators have been summarized in Table 1 to facilitate direct comparison of the differences between the two. The two models, VAE and VAE-shift, have their own advantages and disadvantages in automatic evaluation indicators. Specifically, the VAE model performs better in the BLEU-4 and METEOR indicators, indicating that the n-gram coverage and semantic similarity of the sentences it generates are high, and it can generate sentences of good quality and semantically similar to the reference sentences. At the same time, the VAE model is also slightly higher than the VAE-shift in the recall rate of the ROUGE indicator, indicating that it is slightly better in the completeness of information capture. However, the VAE-shift model comprehensively surpasses the VAE model in the accuracy of the ROUGE indicator, especially in ROUGE-L, where a higher f-value is achieved, indicating that the coherence and fluency of the generated sentences are better. In addition, the VAE-shift model also significantly outperforms the VAE model in the CIDEr indicator, achieving a high score of 0.8127, indicating that the sentences it generates are semantically closer to the manually annotated reference sentences, which is particularly important considering the importance of semantic accuracy in calligraphy evaluation. Based on these indicators, although the VAE model has an advantage in some indicators, the VAE-shift model performs better in semantic accuracy and coherence, and may be more suitable for calligraphy evaluation tasks.

[0128] Example 3

[0129] Reference Figure 4 As an embodiment of the present invention, a calligraphy evaluation system based on a variational autoencoder with category constraints is provided, including a data generation module, a rating module, and an evaluation module.

[0130] The data generation module is used to generate latent variable representations of Chinese character images and corresponding evaluation texts through feature extraction and compression of images and texts. The rating module is used to construct consistency constraints and semantic associations between images and texts using category offsets and bidirectional long short-term memory network decoders, and to reflect different levels of Chinese character writing quality by adjusting the mean of latent variables. The evaluation module is used to constrain latent variables to generate evaluations using the KL divergence between the posterior distribution and the prior distribution.

[0131] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0132] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0133] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0134] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc. It should be noted that the above embodiments are only used to illustrate the technical solution of the present invention and are not limited. Although the present invention is described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.

[0135] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A calligraphy evaluation method based on a category-constrained variational autoencoder, characterized in that: include: By extracting and compressing the features of images and texts, latent variable representations of Chinese character images and corresponding evaluation texts are generated; The class offset and bidirectional long short-term memory network decoder are used to construct the consistency constraints and semantic relevance between images and texts, and the different levels of Chinese character writing quality are reflected by adjusting the mean of latent variables. The KL divergence between the posterior distribution and the prior distribution is used to constrain the latent variable generation evaluation.

2. The calligraphy evaluation method based on the category-constrained variational autoencoder according to claim 1, characterized in that: The method of generating a latent variable representation of a Chinese character image and a corresponding evaluation text by extracting and compressing features of an image and text includes: an image encoder receiving a Chinese character image as input, using a residual network as a backbone network, extracting features and compressing dimensions of the input image, and obtaining a compressed representation of the image h img h img =ResNet50(img) The image encoder uses two independent fully connected layers FC to transform h img Mapped to the mean μ and logarithmic standard deviation logσ of the latent variable z, expressed as: μ img =FC μ (h img ) logσ img =FC σ (h img ) Among them, μ img and logσ img They respectively represent the mapping of the compressed image to the mean and logarithmic standard deviation of the latent variable z.

3. The calligraphy evaluation method based on the category-constrained variational autoencoder according to claim 2, characterized in that: The method of generating latent variable representations of Chinese character images and corresponding evaluation texts by extracting and compressing features of images and texts includes adjusting latent variables by using category information predefined in a data set after encoding the input image by an image encoder to obtain latent variable representations, introducing prior knowledge, and dividing all Chinese characters into three levels of good, medium, and poor according to writing quality when preparing the data set, and assigning three category labels of 0, 1, and 2 respectively. For each category, a different offset shift is set, and the latent variable encodings of different categories are shifted in different directions by subtracting the offset of the corresponding category from the mean vector of the latent variable, which is expressed as:

4. The calligraphy evaluation method based on the category-constrained variational autoencoder according to claim 3, characterized in that: The method of using the category offset and the bidirectional long short-term memory network decoder to construct the consistency constraint and semantic association between the image and the text includes: the text encoder receives the evaluation text corresponding to the Chinese character image as input, uses the LSTM network to extract features and semantically encode the input evaluation, and obtains the compressed representation h of the text. text , expressed as: h text =LSTM(text) The text encoder uses two independent fully connected layers to map htext to the latent variable mean μ text and logarithmic standard deviation, at the mean parameter μ text The category shift that reflects the quality level of Chinese characters is incorporated into the image to achieve the consistency constraint of the text information, and the sampling result z is generated by reparameterization. text , expressed as: μ text =FC μ (h text ) logσ text =FC σ (h text ) 5. The calligraphy evaluation method of the variational autoencoder based on category constraints as claimed in claim 4, characterized in that: The method of using the category offset and the bidirectional long short-term memory network decoder to construct the consistency constraint and semantic association between the image and the text also includes using the same bidirectional long short-term memory network as a decoder to respectively generate the potential representation z of the image encoder and the text encoder. img and z text Decode and get dec img and dec text ; z img Input into the BiLSTM decoder, after multiple layers of nonlinear transformation, the image decoding result dec is obtained img , expressed as: dec img =BiLSTM(z img ) z text Input into the same BiLSTM decoder to get the text decoding result dec text , expressed as: dec text =BiLSTM(z text ) Using two independent fully connected layers, dec img and dec text Mapped to the word table space, and the generation probability of each word is obtained through the normalization function, expressed as: p(x t |z img )=softmax(FC(dec img )) p(x t |z text )=softmax(FC(dec text )) where x t Represents the tth word generated.

6. The calligraphy evaluation method based on the category-constrained variational autoencoder according to claim 5, characterized in that: The method of adjusting the mean of latent variables to reflect different levels of Chinese character writing quality includes constructing training losses including reconstruction loss, image latent representation loss and text latent representation loss; The reconstruction loss measures the difference between the decoder generated results and the real text. img and dec text With real text input text The cross entropy loss between is expressed as: in, represents the cross entropy loss function, expressed as Among them, p(x) represents the true value, which means the real evaluation text in this project, and q(x) represents the predicted value; The image latent representation loss is expressed as: Among them, q(z) represents the posterior distribution and p(z) represents the prior distribution.

7. The calligraphy evaluation method based on the category-constrained variational autoencoder according to claim 6, characterized in that: The KL divergence between the posterior distribution and the prior distribution is used to constrain the latent variable generation evaluation, including the image potential representation loss used to constrain the potential representation generated by the image encoder, introduce a prior distribution, and minimize z img The KL divergence between the prior distribution is expressed as: Llatent img =D KL (q(z img ∣x img )||p(z img )) Among them, q(z img |x img ) represents the posterior distribution generated by the image encoder, p(z img ) represents the prior distribution generated by the image encoder, which ensures that the generated z img Conform to the preset prior distribution; The text potential representation loss is used to constrain the potential representation z generated by the text encoder text , expressed as: Llatent text =D KL (q(z text ∣x text )||p(z text )) Among them, q(z text |x text ) represents the posterior distribution generated by the text encoder, p(z text ) represents the prior distribution generated by the text encoder, minimizing the text potential representation loss to ensure the generated z text In line with the preset prior distribution, The overall loss function of the model is the weighted sum of reconstruction loss, image potential representation loss, and text potential representation loss, expressed as: Among them, λ is the weight coefficient of the reconstruction loss term. By balancing the three loss functions, the model simultaneously learns the potential representation of images and texts to generate evaluation text.

8. A system for calligraphy evaluation using the variational autoencoder based on category constraints as claimed in any one of claims 1 to 7, characterized in that: Including data generation module, rating module, and evaluation module; The data generation module is used to generate latent variable representations of Chinese character images and corresponding evaluation texts by extracting and compressing features of images and texts; The rating module is used to construct the consistency constraint and semantic relevance between the image and the text by using the category offset and the bidirectional long short-term memory network decoder, and to reflect the different levels of Chinese character writing quality by adjusting the mean of the latent variable; The evaluation module is used to utilize the KL divergence between the posterior distribution and the prior distribution to constrain latent variables to generate evaluations.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the calligraphy evaluation method based on the category-constrained variational autoencoder according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the calligraphy evaluation method based on a category-constrained variational autoencoder according to any one of claims 1 to 7 are implemented.

Images