Evaluation method and device based on intelligent ink image, electronic equipment and storage medium

Abstract

The invention relates to an evaluation method and device based on an intelligent ink image, electronic equipment and a storage medium. The evaluation method comprises the following steps: step 1, performing style migration training on a ChipGAN model by utilizing a content data set of a preset number of pre-acquired images subjected to feature classification and style data sets of ink representations of different artists; and step 2, based on the structural similarity index and the color difference, combining subjective visual evaluation of a user on the migrated image and the like. The invention further provides an intelligent ink image evaluation device, electronic equipment and a computer readable storage medium. The method has the advantages that the problems that the difference between an existing image quality evaluation index and a human subjective visual evaluation effect is large, the weight of each feature of the image in evaluation cannot be well balanced, and the influence of the particularity of the ink style on evaluation is ignored are at least partially solved; according to the evaluation device, the academic research of intelligent ink image evaluation is applied to practice, and the multi-element appreciation of the public on the generative ink art is met.

Description

Technical Field

[0001] The present invention relates to the technical field of image evaluation methods and devices, and in particular to an intelligent ink-wash image evaluation method, device, electronic device, and storage medium. Background Art

[0002] In recent years, with the continuous development of new media technologies, art and technology are integratively developing. How to present traditional Chinese ink-wash paintings in a digital form has attracted people's attention. Among them, style transfer refers to the process in which a computer reconstructs an image by extracting the texture of one image and the style features of another image. Ink-wash style transfer is a typical application in the field of intelligent ink-wash, and the application of intelligent ink-wash images has gradually come into the public eye and needs to be appreciated by the public.

[0003] Some Chinese scholars have conducted research on ink-wash style transfer based on artificial intelligence technology, resulting in the Chinese Ink Wash Painting Style Transfer Generative Adversarial Network (ChipGAN), the "Daozi" intelligent painting system of the Future Laboratory of Tsinghua University, and the "immersive digital ink-wash painting" application tool supported by Tencent Multimedia Laboratory. These research or applications have verified the effectiveness of their methods and the feasibility of their applications through general image quality evaluation indicators. However, people's perception of art is different. Especially at present, the society still has different opinions on generative art. Whether it is the research on ink-wash style transfer based on artificial intelligence-related theories or the above-mentioned generative ink-wash systems, there is no exclusive intelligent ink-wash art evaluation system.

[0004] Currently, there are at least the following disadvantages and deficiencies in the existing technologies: 1. The existing image quality evaluation indicators are quite different from the subjective visual evaluation results of people, and cannot well balance the weights of various features of the image in the evaluation; the existing indicators focus more on the evaluation of the content of the generated image and ignore the influence brought by the ink-wash style; 2. Based on the evaluation of intelligent ink-wash images in academic research, there is no complete set of intelligent ink-wash image systems or applications developed around the evaluation, which cannot meet the diverse appreciation needs of the general public.

[0005] In the prior art, for example, Chinese Patent Application No. 202010532759.X discloses a method for automatically generating Chinese ink paintings based on the Generative Adversarial Network (GAN), which includes the following steps: 1) Obtain a Chinese ink painting image dataset and preprocess the Chinese ink painting images; 2) Decompose the preprocessed Chinese ink painting dataset into datasets of different categories; 3) Use the non-local means denoising algorithm to denoise the dataset to obtain a feature dataset; 4) Establish a Generative Adversarial Network (GAN) using the training dataset and determine the input image size; 5) Input the feature dataset into the Generative Adversarial Network (GAN) for training to obtain a trained GAN neural network model; 6) Input the category label data into the trained GAN neural network model to automatically generate Chinese ink paintings corresponding to the labels. The present invention solves the problems of cumbersome manual operations and low creation efficiency in traditional design methods.

[0006] Again, for example, Chinese Patent Application No. 202110520598.7 discloses a method, system, computer device, and storage medium for Chinese ink painting art style conversion. The method includes: obtaining a dataset and preprocessing the dataset to obtain a preprocessed dataset; obtaining a training set based on the preprocessed dataset; establishing an asymmetric cycle-consistent generative adversarial network model, which includes an asymmetric structure mechanism and a saliency edge loss function on the basis of the CycleGAN framework; using the training set to train the asymmetric cycle-consistent generative adversarial network model; inputting the to-be-tested real natural image into the trained asymmetric cycle-consistent generative adversarial network model to obtain a Chinese ink painting image, thereby realizing Chinese ink painting style conversion. This invention overcomes the adverse effects brought by the asymmetry between image domains in the Chinese ink painting art conversion task and achieves a Chinese ink painting style conversion effect with higher quality.

[0007] Furthermore, for example, Chinese Patent Application No. 202410145233.4 discloses an image style transfer method applied to Chinese landscape paintings. First, construct a training set and a test set from the image data composed of unpaired real landscape images and Chinese landscape painting images; then, construct a style transfer model, extract features from the real landscape images in the input training set through a residual attention generator to generate corresponding fake images; next, input the fake images and the real landscape images into a pre-trained text-image encoder to encode them into semantic vectors, and then calculate the semantic loss function by the semantic constraint module introduced by the model; furthermore, transmit the calculated semantic loss function to the style transfer model, optimize the residual attention generator through backpropagation, update the model parameters, and iteratively train until the model converges; finally, use the test set to test the trained style transfer model. The invention effectively solves the problems of unclear details and fuzzy semantic features in the Chinese landscape painting style transfer task.

[0008] None of the above-mentioned invention patent applications involve a complete set of intelligent ink painting image systems or evaluation methods and devices centered around intelligent ink painting images, which cannot meet the diverse appreciation needs of the general public. Summary of the Invention

[0009] The purpose of the present invention is to provide an intelligent ink painting image-based evaluation method, device, electronic device, and storage medium in view of the deficiencies of the prior art.

[0010] An intelligent ink painting image-based evaluation method includes the following steps:

[0011] In step 1, the ChipGAN model is trained for style transfer using a pre-acquired content data set of a predetermined number of images classified by features and a style data set of ink painting masterpieces by different artists.

[0012] In step 2, based on the Structural Similarity Index (SSIM) and Hue, Saturation, Value (HSV), combined with the user's subjective visual evaluation of the transferred image, the weights of each feature component in the evaluation indexes based on the structural similarity index and color difference value are optimized, and organically integrated to generate an evaluation system suitable for intelligent ink painting images.

[0013] In step 3, the intelligent ink painting image evaluation method before and after optimization is compared to verify the effectiveness of the present method.

[0014] Furthermore, in step 1, the specific steps for classifying the content data set and the style data set are as follows:

[0015] In step 1.1, the content data set is roughly classified according to the subjective visual perception of the structural complexity of the image and the weather light and dark characteristics. For example, selecting photos of the Forbidden City or documentary video frames as the content data set, the panoramic view of the building complex can be a complex structure, while the distant view of the corner tower is a single structure. Then, the photos of several common different weathers such as sunny, rainy, snowy, and night are classified, divided into complex-sunny, complex-rainy, complex-snowy, complex-night, single-sunny, single-rainy, single-snowy, single-night, a total of 8 classifications.

[0016] In step 1.2, the gray-level co-occurrence matrix is used to calculate the average entropy value of the images under each visual classification to verify the correctness of the structural classification in step 1, and the average contrast of the images under each visual classification is calculated to verify the correctness of the weather classification in step 1. Finally, each of the 8 classifications contains 500 pictures.

[0017] Step 1.3 Use different ink-wash arts such as Huang Binhong, Qi Baishi, Xu Beihong, Zhang Yu, and Particle Color Ink as the first five style datasets, and select 60 representative works of each of the above five ink-wash art styles for mixing as the sixth style dataset. Finally, each of the six style datasets contains 300 pictures;

[0018] Step 1.4 Perform style transfer training on ChipGAN using the classified content dataset and style dataset, including: inputting the classified eight types of content datasets of complex-sunny, complex-rainy, complex-snowy, complex-night, single-sunny, single-rainy, single-snowy, and single-night and six groups of style datasets of Huang Binhong, Qi Baishi, Xu Beihong, Zhang Yu, Particle Color Ink, and mixed into ChipGAN in a permutation and combination manner for transfer training to obtain corresponding ink-wash style images; evaluate the generated ink-wash style images.

[0019] Furthermore, in Step 2, use SSIM to judge the structural similarity between the generated ink-wash image and the original content style image and use HSV color difference to judge the semantic similarity between the generated ink-wash image and the original content style image, including: the influence of different features of the content image on SSIM during evaluation and the influence of different features of the style image on HSV color difference during evaluation; analyze the influence degree of each feature component of SSIM and HSV color difference on evaluation under different features.

[0020] Furthermore, in Step 2, use the subjective visual perception of users to evaluate the similarity between the generated ink-wash image and the original content style image: use SSIM to judge the structural similarity between the generated ink-wash image and the original content style image and use HSV color difference to judge the semantic similarity between the generated ink-wash image and the original content style image.

[0021] Furthermore, in Step 2, the influence degree of each feature component of SSIM and HSV color difference on evaluation under different features includes: list possible combinations of feature component weights, perform feature extraction and re-transfer on different classification datasets; and optimize the weights of each component of SSIM and the weights of each component of HSV color difference using the normalized results of subjective visual evaluation after re-transfer.

[0022] Furthermore, in Step 2, the selection steps for the combination of the two index feature weights of SSIM and HSV are as follows:

[0023] Step 2.1 SSIM has three components: luminance similarity (l), contrast similarity (c), and structural similarity (s), and there are several weight distribution tendencies such as l>c>s, l>s>c, c>l>s, c>s>l, s>c>l, s>l>c;

[0024] Step 2.2 Select the weight preference suitable for different datasets according to the influence degrees of the three components on the SSIM evaluation. For example, the Forbidden City complex photographed on a sunny day is a content dataset, and Huang Binhong's landscape paintings are a style dataset. At this time, the influence of the original image structure on its migration effect is relatively large, and the influence of the weather on its migration effect is relatively small. The influence degrees of the various components in SSIM on the intelligent ink painting evaluation are: s > l > c;

[0025] Step 2.3 Denote the sum of the three components as 1, and further allocate weights according to the weight preference selected for the corresponding dataset. For example, according to the weight combinations of the l, s, and c components in SSIM in Step 2.1, they are: 3:5:2, 4:5:1, 3:6:1, 4:6:0, 2:7:1, 3:7:0, 2:8:0, 1:9:0;

[0026] Step 2.4 Use the gray-level co-occurrence matrix to extract the texture of each feature of the content image according to the weights in Step 2.3, and perform migration with the original style dataset again;

[0027] The HSV color difference has three components: hue difference H, saturation difference S, and brightness difference V. Perform operations similar to SSIM. Finally, the HSV feature maps of the style image need to be extracted according to the weight combinations of the features that can be, and then perform migration with the original content dataset again;

[0028] Step 2.6 Perform normalization processing on the subjective evaluation results after the second migration. The specific steps are as follows:

[0029] Step 2.6.1 Subjectively evaluate the results after the second migration in Step 2.4, and find out the weight combination of l, s, and c corresponding to the optimal migration effect;

[0030] Step 2.6.2 Subjectively evaluate the results after the second migration in Step 2.5, and find out the weight combination of H, S, and V corresponding to the optimal migration effect;

[0031] Step 2.6.3 Calculate the normalization results of the three feature components of SSIM and the three feature components of the HSV color difference respectively according to the following formula (1):

[0032]

[0033] Where: x is the l, s, c weight value or H, S, V weight value corresponding to the optimal migration effect with the most votes from the evaluators in each group of subjective evaluations; S is the standard deviation of the l, s, c weight value or H, S, V weight value corresponding to the optimal migration effect found by each evaluator in each group of subjective evaluations; μ is the average of the l, s, c weight value or H, S, V weight value corresponding to the optimal migration effect found by each evaluator in each group of subjective evaluations; xscale is the normalized value of x, that is, the final weight corresponding to each feature under different categories after calculation;

[0034] Step 2.7: Substitute the weights calculated in step 2.6 to calculate the optimized SSIM and HSV color difference values:

[0035] Step 2.7.1: Substitute the x corresponding to the brightness, contrast, and structural similarity of a certain content image classification calculated by formula (1) into scale Denoted as α, β, γ, as the weights of l, c, s respectively, the three weight values ​​are respectively substituted into the following formula (2), and the evaluation value of SSIM weight optimization under this category is calculated, where: i is the structural complexity of the content graph. When i = 1, it means that the structure of the content graph of this category is complex and the average entropy value is large. When i = 2, it means that the structure of the content graph of this category is simple and the average entropy value is small. j represents the weather of the content graph. The j value of 1, 2, 3, 4 represents the situation of sunny, rainy, snowy and night in the content graph respectively. Among them, when Huang Binhong's landscape painting is used as the style data set or the five styles of Huang Binhong, Qi Baishi, Xu Beihong, Zhang Yu and granular ink painting are mixed, it is necessary to satisfy γ1+γ2=1, α1+α2+α3+α4=1, β1+β2+β3+β4=1. When Qi Baishi, Xu Beihong, Zhang Yu and granular ink painting are used as the style data set respectively, it is necessary to satisfy β1+β2=1, α1+α2+

[0036] Step 2.7.2 Classify the hue, saturation, and brightness of a style map calculated by formula (1)

[0037]

[0038] The difference in x scale Let λ, μ, and θ be the weights of H, S, and V respectively. Substitute the three weight values ​​into the following formula (3) to calculate the optimized evaluation value of HSV color difference weight under this classification. Here, k represents different ink painting styles. k = 1, 2, 3, 4, 5, and 6 represent the styles of Huang Binhong, Qi Baishi, Xu Beihong, Zhang Yu, particle color ink, and mixed ink painting respectively. r and h are the base radius and height of the HSV cone color space respectively, which are generally fixed values.

[0039]

[0040] Step 2.7.3 According to the following formula (4), fuse the evaluation indicators, and calculate the absolute values of the optimized weighted SSIM (between 0 and 1, the closer to 1, the more similar the migrated image is to the content image, and the better the effect) and the HSV color difference (between 0 and 1, the closer to 0, the smaller the difference between the migrated image and the style image, and the better the effect). The larger the value, the more similar the generated image is to the original image:

[0041]

[0042] Furthermore, in Step 3, compare the intelligent ink painting image evaluation methods before and after optimization to verify the effectiveness of this method, including: using the Pearson linear correlation coefficient, Spearman rank correlation coefficient, Kendall rank correlation coefficient, and root mean square error and other evaluation indicators in the Image Quality Assessment of Video Quality Experts Groups (abbreviation: IQA method in VQEG) for image quality evaluation, combined with the user's subjective visual evaluation results to verify the optimized intelligent ink painting image evaluation indicators, including the following steps:

[0043] Step 3.1 Verify the Pearson linear correlation coefficient of the calculated subjective and objective scores. If the absolute value is closer to 1, it indicates a stronger correlation between the subjective and objective evaluations;

[0044] Step 3.2 Calculate the Spearman rank correlation coefficient of the subjective and objective scores. Similarly, if the absolute value is closer to 1, it indicates a stronger correlation between the subjective and objective evaluations;

[0045] Step 3.3 Calculate the Kendall rank correlation coefficient of the subjective and objective scores. Similarly, if the absolute value is closer to 1, it indicates a stronger correlation between the subjective and objective evaluations;

[0046] Step 3.4 Calculate the root mean square error between the subjective and objective scores. The smaller the value, the more accurate the prediction.

[0047] The present invention further provides an intelligent ink painting image evaluation device, including: a style migration module, a subjective and objective evaluation module, and a verification module / evaluation comparison module; the style migration module is used to migrate the input image according to different ink painting styles by using the trained ChipGAN model; the subjective and objective evaluation module is used to evaluate the generated ink painting image by using the optimized intelligent ink painting image evaluation method; the verification module / evaluation comparison module is used to compare and verify the optimized evaluation method with the user's subjective visual evaluation results, providing a reference for the user to evaluate the intelligent ink painting image.

[0048] The present invention further provides an electronic device based on the intelligent ink painting image evaluation method and the intelligent ink painting image evaluation device. The electronic device includes: a memory and a processor. A computer program capable of running on the processor is stored in the memory. When the processor executes the computer program, the specific steps of the intelligent ink painting image evaluation method are implemented.

[0049] The present invention further provides a computer-readable storage medium used based on the intelligent ink painting image evaluation method and the intelligent ink painting image evaluation device. The computer-readable storage medium stores machine-executable instructions. When the computer running instructions are called and run by the processor, the computer running instructions start the processor to run the specific steps of the evaluation method.

[0050] The intelligent ink painting image evaluation method, device, electronic device and storage medium of the present invention have the following remarkable and superior technical effects:

[0051] 1. By adopting the intelligent ink painting image evaluation method of the present invention, at least some of the problems that the existing image quality evaluation indexes have a large difference from the human subjective visual evaluation effect, cannot well balance the weights of various features in the evaluation, and ignore the influence of the particularity of the ink painting style on the evaluation are solved.

[0052] 2. The intelligent ink painting image evaluation method of the present invention classifies and transfers and trains the content data set and the style data set according to different features, and selects the index to be optimized and combines the human subjective visual evaluation results to find the weight tendency of each feature when evaluating the intelligent ink painting image under various classifications, providing a foundation for promoting the unity of the index and the human subjective visual evaluation, and paying attention to the application of the image content and style features in the evaluation.

[0053] 3. The intelligent ink painting image evaluation method of the present invention lists the possible feature weight combinations of each group of data sets according to the image feature weight tendency, performs feature extraction and re-transfer training on the original content and style images, and through the normalization processing of the subjective evaluation results of the generated ink painting images, classifies according to different content and style features, and precisionizes the weights of each component of the index to be optimized.

[0054] 4. The intelligent ink painting image evaluation method of the present invention combines the evaluation of content similarity and style difference of intelligent ink painting images. The content similarity is manifested in the similarity of the structural features between the intelligent ink painting image and the original content image. Among them, the better the effect, the larger the optimized SSIM value. The style difference is manifested in the color difference between the intelligent ink painting image and the original style image. The better the effect, the smaller the optimized HSV color difference value. Combining the two evaluation methods means that the greater the absolute value of the two, the better the overall generation effect. Through the IQA method in VQEG, the comparison and verification of the indicators before and after optimization and the subjective visual evaluation results are carried out, and the applicable intelligent ink painting image evaluation indicators are given for different dataset classifications respectively.

[0055] 5. The intelligent ink painting image evaluation device of the present invention applies the academic research on the evaluation of intelligent ink painting images to practice, forms a system dedicated to the evaluation of intelligent ink painting images, and meets the diverse appreciation of the public for generative ink painting art.

[0056] 6. The electronic device for the intelligent ink painting image evaluation method and the intelligent ink painting image evaluation device provided by the present invention can implement each specific step of the evaluation method.

[0057] 7. The computer-readable storage medium for the intelligent ink painting image evaluation method and the intelligent ink painting image evaluation device provided by the present invention stores machine-executable instructions. When the computer-executable instructions are called and run by the processor, the computer-executable instructions start the processor to run each relevant specific step of the intelligent ink painting image evaluation method. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 is a schematic flow chart of the evaluation method of the present invention.

[0059] Figure 2 is a schematic structural diagram of the evaluation device of the present invention.

[0060] Figure 3 is a schematic structural composition diagram of the electronic device for the evaluation method and the evaluation device of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0061] Now in combination with the accompanying Figures 1-3 The specific embodiments of the evaluation method, evaluation device, electronic device and storage medium of the present invention are introduced in detail.

[0062] Embodiment:

[0063] As Figure 1 shown, the evaluation method of the present invention includes the following steps:

[0064] In Step 1, the ChipGAN model is trained for style transfer using a content dataset of a predetermined number of pre-acquired feature-classified images and a style dataset of representative ink paintings by different artists.

[0065] In Step 1, the trained ChipGAN model is used to perform ink painting style transfer training on the dataset classified by multi-scene features respectively.

[0066] In Step 2, based on the Structural Similarity Index (SSIM) and Hue, Saturation, Value (HSV), combined with the user's subjective visual evaluation of the transferred images, the weights of each feature component in the evaluation metrics based on the structural similarity index and color difference values are optimized and organically integrated to generate an evaluation system suitable for intelligent ink paintings.

[0067] In Step 3, the intelligent ink painting evaluation methods before and after optimization are compared to verify the effectiveness of this method.

[0068] As a specific step in the embodiment, in Step 1, the ChipGAN model is trained for style transfer using a content dataset of a predetermined number of pre-acquired feature-classified images and a style dataset of representative ink paintings by different artists: The specific steps for classifying the content dataset and the style dataset are as follows:

[0069] Step 1.1 Roughly classify the content dataset according to the subjective visual perception of the structural complexity of the images and the weather brightness and darkness features. For example, select photos of the Forbidden City or documentary video frames as the content dataset. The panoramic view of the building complex can be a complex structure, while the distant view of the corner tower is a single structure. Then, classify according to photos of several common different weathers such as sunny, rainy, snowy, and night, and divide them into complex-sunny, complex-rainy, complex-snowy, complex-night, single-sunny, single-rainy, single-snowy, single-night, a total of 8 classifications.

[0070] Step 1.2 Use the gray-level co-occurrence matrix to calculate the average entropy value of the images under each visual classification to verify the correctness of the structural classification in Step 1, and calculate the average contrast of the images under each visual classification to verify the correctness of the weather classification in Step 1. Finally, each of the 8 classifications contains 500 pictures.

[0071] Step 1.3 Take the different ink arts such as Huang Binhong, Qi Baishi, Xu Beihong, Zhang Yu, and particle color ink as the first 5 style datasets, and select 60 representative works of each of the above 5 ink art styles for mixing as the 6th style dataset. Finally, each of the 6 style datasets contains 300 pictures.

[0072] Step 1.4 performs style transfer training on the classified content dataset and style dataset using ChipGAN, including: inputting the classified content datasets of 8 categories, namely complex - sunny, complex - rainy, complex - snowy, complex - night, single - sunny, single - rainy, single - snowy, and single - night, and 6 groups of style datasets of Huang Binhong, Qi Baishi, Xu Beihong, Zhang Yu, particle ink painting, and hybrid, into ChipGAN in a permutation and combination manner for transfer training to obtain corresponding ink - wash style images; evaluating the generated ink - wash style images.

[0073] As a specific step in the embodiment, in Step 2, the structural similarity between the generated ink - wash image and the original content image is judged using SSIM, and the semantic similarity between the generated ink - wash image and the original style image is judged using the HSV color difference, including: the influence of different features of the content image on SSIM during evaluation and the influence of different features of the style image on the HSV color difference during evaluation; analyzing the influence degree of each feature component of SSIM and HSV color difference on the evaluation under different features.

[0074] As a specific step in the embodiment, in Step 2, the similarity between the generated ink - wash image and the original content and style images is evaluated using the subjective visual perception of the user; the structural similarity between the generated ink - wash image and the original content image is judged using SSIM, and the semantic similarity between the generated ink - wash image and the original style image is judged using the HSV color difference.

[0075] As a specific step in the embodiment, in Step 2, the influence degree of each feature component of SSIM and HSV color difference on the evaluation under different features includes: listing possible combinations of feature component weights, extracting features and performing re - transfer on different classification datasets; and optimizing the weights of each component of SSIM and the weights of each component of HSV color difference using the normalized results of subjective visual evaluation after re - transfer.

[0076] As a specific step in the embodiment, the selection steps of the weight combination of the two index features of SSIM and HSV in Step 2 are as follows:

[0077] Step 2.1 SSIM has three components: luminance similarity (l), contrast similarity (c), and structural similarity (s), and there are several weight distribution tendencies such as l > c > s, l > s > c, c > l > s, c > s > l, s > c > l, s > l > c.

[0078] Step 2.2 Select the weight preference suitable for different datasets according to the influence degrees of the three components on the SSIM evaluation. For example, the Forbidden City complex photographed on a sunny day is a content dataset, and Huang Binhong's landscape paintings are a style dataset. At this time, the influence of the original image structure on its migration effect is relatively large, and the influence of the weather on its migration effect is relatively small. The influence degrees of the components in SSIM on the intelligent ink painting evaluation are: s > l > c;

[0079] Step 2.3 Denote the sum of the three components as 1, and further allocate weights according to the weight preference selected for the corresponding dataset. For example, according to the weight combinations of the l, s, and c components in SSIM in Step 2.1, they are: 3:5:2, 4:5:1, 3:6:1, 4:6:0, 2:7:1, 3:7:0, 2:8:0, 1:9:0;

[0080] Step 2.4 Use the gray-level co-occurrence matrix to extract the textures of the features of the content image according to the weights in Step 2.3, and perform migration with the original style dataset again;

[0081] The HSV color difference has three components: hue difference (H), saturation difference (S), and brightness difference (V). Perform operations similar to SSIM. Finally, the HSV feature maps of the style image need to be extracted according to the possible feature weight combinations and migrated with the original content dataset again;

[0082] Step 2.6 Perform normalization processing on the subjective evaluation results after the second migration. The specific steps are as follows:

[0083] Step 2.6.1 Subjectively evaluate the results after the second migration in Step 2.4, and find the l, s, c weight combination corresponding to the optimal migration effect;

[0084] Step 2.6.2 Subjectively evaluate the results after the second migration in Step 2.5, and find the H, S, V weight combination corresponding to the optimal migration effect;

[0085] Step 2.6.3 Calculate the normalized results of the three components of SSIM and the three components of the HSV color difference according to the following formula (1):

[0086]

[0087] Where: x is the l, s, c weight value or H, S, V weight value corresponding to the optimal migration effect with the most votes in each group of classification subjective evaluations by the evaluators. S is the standard deviation of the l, s, c weight value or H, S, V weight value corresponding to the optimal migration effect found by each evaluator in each group of classification subjective evaluations. μ is the average value of the l, s, c weight value or H, S, V weight value corresponding to the optimal migration effect found by each evaluator in each group of classification subjective evaluations. xscale is the value of x after normalization, that is, the final weight corresponding to each feature under different classifications after calculation;

[0088] Step 2.7 Substitute the weights calculated in Step 2.6 to calculate the optimized SSIM and HSV color difference values:

[0089] Step 2.7.1 Denote the xscale corresponding to brightness, contrast, and structural similarity under a certain content map classification calculated by formula (1) as α, β, γ, which are used as the weights of l, c, s respectively, and substitute these three weight values

[0090] into the following formula (2) respectively, then calculate the evaluation value of the optimized SSIM weight under this classification. Where: i is the structural complexity of the content map. When i = 1, it means the content map of this classification has a complex structure and a large average entropy value. When i = 2, it means the content map of this classification has a single structure and a small average entropy value. j represents the weather of the content map. The values of j being 1, 2, 3, 4 represent the content map being sunny, rainy, snowy, and night respectively. Among them, when the Huang Binhong landscape painting is the style dataset or a mixture of 5 styles (that is, a mixture of 5 styles of Huang Binhong, Qi Baishi, Xu Beihong, Zhang Yu, and Particle Color Ink in Step 1.3), it is necessary to satisfy γ1 + γ2 = 1, α1 + α2 + α3 + α4 = 1, β1 + β2 + β3 + β4 = 1. When the other 4 art styles are style datasets (that is, when the works of Qi Baishi, Xu Beihong, Zhang Yu, and Particle Color Ink are used as style datasets respectively), it is necessary to satisfy β1 + β2 = 1, α1 + α2 + α3 + α4 = 1, γ1 + γ2 + γ3 + γ4 = 1;

[0091]

[0092] Step 2.7.2 For a certain style map classification, substitute the hue, saturation, and brightness calculated by formula (1)

[0093] The xscales corresponding to the differences are denoted as λ, μ, and θ, which are used as the weights of H, S, and V respectively. Substituting the three weight values into the following formula (3), the optimized evaluation value of the HSV color difference weight under this classification can be calculated. Here, k represents different ink painting styles, where k = 1, 2, 3, 4, 5, 6 represent Huang Binhong's landscape paintings, Qi Baishi's fish and shrimps, Xu Beihong's galloping horse paintings, Zhang Yu's new national trend color ink paintings, particle ink animation style, and mixed ink style respectively. r and h are the bottom radius and height of the HSV conical color space, which are generally fixed values.

[0094]

[0095] Step 2.7.3 According to the following formula (4), fuse the evaluation indicators, and calculate the absolute values of the optimized weight SSIM (between 0 and 1, the closer to 1, the more similar the migrated image and the content image, and the better the effect) and the HSV color difference (between 0 and 1, the closer to 0, the smaller the difference between the migrated image and the style image, and the better the effect). The larger the value, the more similar the generated image and the original image are:

[0096]

[0097] As a specific step in the embodiment, in Step 3, compare the intelligent ink painting image evaluation method before and after optimization to verify the effectiveness of this method, including: using the Pearson linear correlation coefficient, Spearman rank correlation coefficient, Kendall rank correlation coefficient, and root mean square error and other evaluation indicators in the Image Quality Assessment of Video Quality Experts Groups (abbreviation: IQA method in VQEG) in the Visual Quality Experts Group, combined with the user's subjective visual evaluation results, to verify the optimized intelligent ink painting image evaluation indicators, including the following steps:

[0098] Step 3.1 Verify the Pearson linear correlation coefficient of the subjective and objective scores. If the absolute value is closer to 1, it indicates that the correlation between the subjective and objective evaluations is stronger;

[0099] Step 3.2 Calculate the Spearman rank correlation coefficient of the subjective and objective scores. Similarly, if the absolute value is closer to 1, it indicates that the correlation between the subjective and objective evaluations is stronger;

[0100] Step 3.3 Calculate the Kendall rank correlation coefficient of the subjective and objective scores. Similarly, if the absolute value is closer to 1, it indicates that the correlation between the subjective and objective evaluations is stronger;

[0101] Step 3.4 Calculate the root mean square error of the subjective and objective scores. The smaller the value, the more accurate the prediction.

[0102] The present invention further provides an intelligent ink painting image evaluation device based on a generative adversarial network, such as Figure 2As shown, the evaluation device includes a style transfer module S10, a subjective and objective evaluation module S20, and a verification module / evaluation comparison module S30. The style transfer module S10 uses the trained ChipGAN model to transfer the input image according to different ink painting styles. The subjective and objective evaluation module S20 uses the optimized intelligent ink painting image evaluation method to evaluate the generated ink painting image. The verification module / evaluation comparison module S30 compares and verifies the optimized evaluation method with the user's subjective visual evaluation result to provide a reference for the user to evaluate the intelligent ink painting image.

[0103] The present invention also provides an electronic device for the evaluation method and the evaluation device, such as Figure 3 The electronic device includes a memory 41, a processor 40, a bus 42, and a communication interface 43. The communication interface 43 is communicatively connected to the processor 40 and the memory 41 through the bus 42. The processor 40 is configured to execute an executable module stored in the memory 41, such as a computer program.

[0104] In a specific embodiment of the present invention:

[0105] The memory 41 can include both a high-speed random access memory (RAM) and a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 43 (wired or wireless). The system network element can use the Internet, a wide area network, a local area network, a metropolitan area network, etc.

[0106] The bus 42 can be an ISA bus, a PCI bus, or an EISA bus. The bus is divided into an address bus, a data bus, and a control bus. For the sake of representation, Figure 3 only a bidirectional arrow is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0107] The memory 41 is used to store programs. The processor 40 executes the programs after receiving the execution instructions. Some steps of the evaluation method and the functions of some structural components of the evaluation device can be implemented by the processor 40.

[0108] The processor 40 includes, but is not limited to, an integrated circuit chip with signal processing capabilities. During the implementation of the evaluation method, it is completed through the integrated logic circuit of the hardware in the processor 40 or instructions in the form of software. The processor 40 is a general-purpose processor, including a central processing unit (CPU) and a network processor (NP); it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and can implement or execute the specific method steps and logic block diagrams disclosed in the evaluation method and evaluation device of the present invention; the general-purpose processor is a microprocessor, or the processor is any conventional processor; the various steps of the method disclosed in combination with the embodiments of the present invention are directly embodied as being executed by a hardware decoding processor, or are executed by a combination of hardware and software modules in the decoding processor; the software module can be embedded in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is provided in the memory 41, and the processor 40 reads the information in the memory 41 and combines its hardware devices to implement the relevant steps of the evaluation method.

[0109] As described above, the above is only the specific implementation manner of the present invention, but the protection scope of the present invention is not limited to the specific implementation manner. Any changes or substitutions that can be easily thought of by any person skilled in the art within the scope disclosed by the present invention should be covered by the protection scope of the claims of the present invention.

Claims

1. A method for evaluating an intelligent ink image, characterized in that: The following steps are involved: Step 1, using a pre-acquired content dataset of a predetermined number of feature-classified images and a style dataset of representative ink paintings of different artists to perform style transfer training on the ChipGAN model; Step 2: Based on the structural similarity index and color difference, combined with the user's subjective visual evaluation of the migrated image, the weights of each feature component in the evaluation index based on the structural similarity index and color difference value are optimized, and organically integrated to generate an evaluation system suitable for intelligent ink images; Step 3: Compare the Motu intelligent water image assessment method before and after optimization to verify the effectiveness of this method.

2. The intelligent ink image evaluation method according to claim 1, characterized in that: In step 1, the specific steps for classifying the content dataset and the style dataset are as follows: Step 1.1 roughly classifies the content dataset according to the structural complexity of the image and the subjective visual perception of the light and dark characteristics of the weather. For example, photos of the Forbidden City or documentary video frames are selected as the content dataset. Panoramic shots of building complexes can be complex structures, while distant shots of corner towers are single structures. Then, photos of different common weather conditions such as sunny, rainy, snowy, and night are classified into complex-sunny, complex-rainy, complex-snowy, complex-night, single-sunny, single-rainy, single-snowy, and single-night, for a total of 8 categories; Step 1.2: Use the gray-level co-occurrence matrix to calculate the average entropy value of the images under each visual category to verify the correctness of the structural classification in step 1, and calculate the average contrast of the images under each visual category to verify the correctness of the weather classification in step 1. Finally, each of the 8 categories contains 500 images. Step 1.3: Use different ink art works such as Huang Binhong, Qi Baishi, Xu Beihong, Zhang Yu, and Granular Color Ink as the first five style data sets, and select 60 representative works of each of the five styles of ink art to mix as the sixth style data set. Finally, each of the six style data sets contains 300 images. step 1.4 Use the classified content dataset and style dataset to perform style transfer training on ChipGAN, including: The classified 8 content data sets including complex-sunny, complex-rainy, complex-snow, complex-night, single-sunny, single-rainy, single-snow, and single-night and 6 style data sets including Huang Binhong, Qi Baishi, Xu Beihong, Zhang Yu, particle color ink, and mixed were input into ChipGAN in a permutation and combination manner for migration training to obtain the corresponding ink style images; the generated ink style images were evaluated.

3. The intelligent ink image evaluation method according to claim 1, characterized in that: In step 2, SSIM is used to determine the structural similarity between the generated ink image and the original content style map, and HSV color difference is used to determine the semantic similarity between the generated ink image and the original content style map, including: the impact of different features of the content style map on SSIM during evaluation and the impact of different features of the content style map on HSV color difference during evaluation; analyzing the degree of influence of each feature component of SSIM and HSV color difference on the evaluation under different features.

4. The intelligent ink image evaluation method according to claim 1, characterized in that: In step 2, the user's subjective visual experience is used to evaluate the similarity between the generated ink image and the original content style map, including using SSIM to judge the structural similarity between the generated ink image and the original content style map and using HSV color difference to judge the semantic similarity between the generated ink image and the original content style map.

5. The intelligent ink image evaluation method according to claim 1, characterized in that: In step 2, under different features, the influence of each characteristic component of SSIM and HSV color difference on the evaluation includes: listing possible characteristic component weight combinations, performing feature extraction and re-migration on different classification data sets; optimizing the weights of each SSIM component and optimizing the weights of each HSV color difference component using the normalized results of subjective visual evaluation after re-migration.

6. The intelligent ink image evaluation method according to claim 1, characterized in that: In step 2, the steps for selecting the weight combination of the two indicator features SSIM and HSV are as follows: Step 2.1 SSIM has three components: brightness similarity (l), contrast similarity (c), and structure similarity (s). There are several weight distribution tendencies: l>c>s, l>s>c, c>l>s, c>s>l, s>c>l, s>l>c; Step 2.2 According to the influence of the three components on SSIM evaluation, select the weight tendency suitable for different data sets. For example, the Forbidden City complex photographed on a sunny day is the content data set, and Huang Binhong's landscape paintings are the style data set. At this time, the original image structure has a greater impact on its migration effect, and the weather has a smaller impact on its migration effect. The influence of each component in SSIM in intelligent ink evaluation is: s>l>c; In step 2.3, the sum of the three components is recorded as 1. According to the weight tendency selected for the corresponding data set, the weights are further assigned. According to the weight combination of the three components l, s, and c in SSIM in step 2.1, the weight combination is: 3:5:2, 4:5:1, 3:6:1, 4:6:0, 2:7:1, 3:7:0, 2:8:0, 1:9:0; Step 2.4 uses the gray-level co-occurrence matrix to extract the texture of each feature of the content image according to the weights in step 2.3, and migrates them again with the original style dataset; Step 2.5: HSV color difference has three components: hue difference H, saturation difference S, and brightness difference V. Operations similar to SSIM are performed. Finally, the HSV feature map of the style map needs to be extracted according to the possible feature weight combination and migrated again with the original content dataset; Step 2.6 normalizes the subjective evaluation results after re-transfer. The specific steps are as follows: Step 2.6.1: Conduct a subjective evaluation on the results of the second migration in step 2.4 to find the weight combination of l, s, and c corresponding to the optimal migration effect; Step 2.6.2: Conduct a subjective evaluation on the results of the second migration in step 2.5 to find the H, S, and V weight combination corresponding to the optimal migration effect; Step 2.6.3 calculates the normalized results of the three characteristic components of SSIM and the three characteristic components of HSV color difference respectively according to the following formula (1): Where: x is the l, s, c weight value or H, S, V weight value corresponding to the optimal migration effect with the most votes from the evaluators in each group of subjective evaluations; S is the standard deviation of the l, s, c weight value or H, S, V weight value corresponding to the optimal migration effect found by each evaluator in each group of subjective evaluations; μ is the average of the l, s, c weight value or H, S, V weight value corresponding to the optimal migration effect found by each evaluator in each group of subjective evaluations; xscale is the normalized value of x, that is, the final weight corresponding to each feature under different categories after calculation; Step 2.7 brings in the weights calculated in step 2.6 and calculates the optimized SSIM and HSV color difference values, including: Step 2.7.1: Substitute the x corresponding to the brightness, contrast, and structural similarity of a certain content image classification calculated by formula (1) into scale Denoted as α, β, γ, as the weights of l, c, s respectively, the three weight values ​​are respectively substituted into the following formula (2), and the evaluation value of SSIM weight optimization under this category is calculated, where: i is the structural complexity of the content graph. When i = 1, it means that the structure of the content graph of this category is complex and the average entropy value is large. When i = 2, it means that the structure of the content graph of this category is simple and the average entropy value is small. j represents the weather of the content graph. The j values ​​of 1, 2, 3, and 4 represent the content graph being sunny, rainy, snowy, and night, respectively. In the case of Huang Binhong landscape painting as the style dataset or a mixture of five styles including Huang Binhong, Qi Baishi, Xu Beihong, Zhang Yu and Granule ink painting, it is necessary to satisfy γ1+γ2=1, α1+α2+α3+α4=1, β1+β2+β3+β4=1. When Qi Baishi, Xu Beihong, Zhang Yu and Granule ink painting are used as the style datasets, it is necessary to satisfy β1+β2=1, α1+α2+Step 2.7.2 Classify the hue, saturation and brightness of a style map calculated by formula (1). The difference in x scale Let λ, μ, and θ be the weights of H, S, and V respectively. Substitute the three weight values ​​into the following formula (3) to calculate the optimized evaluation value of HSV color difference weight under this classification. Here, k represents different ink painting styles. k = 1, 2, 3, 4, 5, and 6 represent the styles of Huang Binhong, Qi Baishi, Xu Beihong, Zhang Yu, particle color ink, and mixed ink painting respectively. r and h are the base radius and height of the HSV cone color space respectively, which are generally fixed values. Step 2.7.3: According to the following formula (4), the evaluation indicators are integrated to calculate the absolute value of the SSIM and HSV color differences after optimizing the weights. The larger the value, the more similar the generated image is to the original image:

7. The intelligent ink image evaluation method according to claim 1, characterized in that: In step 3, the intelligent ink image evaluation method before and after optimization is compared to verify the effectiveness of the method, including: using the Pearson linear correlation coefficient, Spearman rank correlation coefficient, Kendall rank correlation coefficient and root mean square error and other evaluation indicators used for image quality evaluation in the visual quality expert group combined with the user's subjective visual evaluation results to verify the optimized intelligent ink image evaluation indicators, specifically: Step 3.1 Verify and calculate the Pearson linear correlation coefficient of the subjective and objective scores. The closer its absolute value is to 1, the stronger the correlation between the subjective and objective evaluations. Step 3.2 Calculate the Spearman rank correlation coefficient of the subjective and objective scores. Similarly, if its absolute value is closer to 1, it means that the correlation between subjective and objective evaluations is stronger; Step 3.3 Calculate the Kendall rank correlation coefficient between the subjective and objective scores. Similarly, if its absolute value is closer to 1, it means that the correlation between the subjective and objective evaluations is stronger; Step 3.4 calculates the root mean square error between the subjective score and the objective score. The smaller the value, the more accurate the prediction.

8. An intelligent ink image evaluation device, comprising: Style transfer module, subjective and objective evaluation module, and verification module / evaluation comparison module; The style transfer module is used to use the trained ChipGAN model to transfer the input image according to different ink styles; the subjective and objective evaluation module is used to evaluate the generated ink images using the optimized intelligent ink image evaluation method; the verification module / evaluation comparison module is used to compare and verify the optimized evaluation method with the user's subjective visual evaluation results, providing a reference for users to evaluate intelligent ink images.

9. An electronic device based on an intelligent ink image evaluation method and an intelligent ink image evaluation device, comprising: A memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the computer program, the specific steps of the intelligent ink image evaluation method are implemented.

10. A computer-readable storage medium based on an intelligent ink image evaluation method and an intelligent ink image evaluation device, wherein the computer-readable storage medium stores machine executable instructions. When the computer execution instructions are called and executed by a processor, the computer execution instructions start the processor to execute the specific steps based on the intelligent ink image evaluation method.

Images