A style transfer image quality evaluation method

Abstract

The application discloses a style transfer image quality evaluation method, which comprises the following steps: inputting content and result images into a content preservation quality prediction network for training, outputting a content preservation quality score and a content preservation weight vector, and obtaining a content preservation quality prediction network training model; inputting style and result images into a style similarity quality prediction network for training, outputting a style similarity quality score and a style similarity weight vector, and obtaining a style similarity quality prediction network training model; inputting the content preservation quality score and the weight vector and the style similarity quality score and the weight vector into an overall visual quality prediction network for training at the same time, outputting an overall visual quality score, and obtaining an overall visual quality prediction network training model; and obtaining the content preservation quality score, the style similarity quality score and the overall visual quality score of the result image by using the three training models during testing.

Description

Technical Field

[0001] This invention relates to an image quality assessment technique, and more particularly to a style transfer image quality assessment method. Background Technology

[0002] Arbitrary style transfer of images is an interesting and practical research area at the intersection of the arts and technology industry and computer vision. It aims to apply arbitrary types of artistic style information to an input image to create a novel artwork. As a unique technique for artistic creation, arbitrary style transfer of images is becoming increasingly popular in many computer vision and image processing tasks, such as image super-resolution, fashion image processing, and generative adversarial learning.

[0003] In recent years, thanks to advancements in deep learning technology, significant progress has been made in the field of arbitrary style transfer of images. However, due to the complexity of evaluation rules and the diversity of subjective preferences, research on how to accurately and objectively evaluate the quality of images generated by different style transfer algorithms remains lacking. Surveys indicate that, apart from a few quantitative indicators, almost all research focuses on evaluating the quality of arbitrarily style-transferred images through large-scale user studies. However, these large-scale user studies are impractical due to their high cost and the significant manpower and time required. Furthermore, some researchers have focused on measuring specific attributes for quantitative comparison, such as running speed, memory usage, and operational control. However, these specific attributes are not suitable for measuring the effectiveness of style transfer results because they have a weak correlation with human visual perception. Therefore, it is essential to research and design image quality evaluation methods to predict the quality of arbitrarily style-transferred images, which has significant implications for future research on arbitrary style transfer of images.

[0004] Currently, there are still several problems in the quality assessment of images with arbitrary style transfer:

[0005] First, while traditional image quality assessment research has made significant progress, it is not suitable for evaluating the perceptual quality of images with arbitrary style transfer. This is because these image quality assessment strategies lack best-practice images for comparison and neglect the specific features of images with arbitrary style transfer, such as semantic structure, style patterns, and overall aesthetics. Therefore, it is necessary to design a specific assessment method that can fully consider the style and content information in the original image and focus on mining specific quality factors.

[0006] Second, while existing methods for evaluating the quality of arbitrary style transfer images are effective, they rely on traditional shallow, handcrafted features and directly use cosine similarity to measure the quality factor. This approach is highly dependent on the overall orientation of feature points and ignores the numerical values ​​of the features. Recently, deep learning-based models have demonstrated superior performance compared to these traditional methods due to their powerful feature extraction capabilities. Therefore, utilizing deep neural networks to improve the robustness of arbitrary style transfer image quality evaluation is a promising alternative.

[0007] Third, existing quality assessment metrics for arbitrary style transfer images, despite incorporating multiple quality factors, suffer from coarse and simplistic final quality pooling processes. This simplistic pooling approach fails to consider the influence of style type information, severing the complex relationship between final quality and different quality factors. It implies that the importance of quality factors for image pairs with different style and content types is predetermined. Therefore, it is essential to research a network model that can understand style image types to simulate human perception, and then use appropriate rules to guide the pooling of different quality factors. Summary of the Invention

[0008] The technical problem to be solved by the present invention is to provide a style transfer image quality evaluation method that can effectively improve the consistency between the objective evaluation results and the subjective perception results of human beings in three aspects: content preservation, style similarity and overall visual.

[0009] The technical solution adopted by this invention to solve the above-mentioned technical problems is: a style transfer image quality evaluation method, characterized by including the following steps:

[0010] Step 1: Construct three deep neural networks: The first deep neural network is a content preservation quality prediction network, which extracts features from both the content image and the result image, then fuses the features of the content image and the result image, and obtains a content preservation quality score to measure the structural content similarity between the content image and the result image, and a content preservation weight vector to balance the importance of content preservation in the overall visual quality. The second deep neural network is a style similarity quality prediction network, which extracts features from both the style image and the result image, then fuses the features of the style image and the result image, and obtains a style similarity quality score to measure the style style similarity between the style image and the result image, and a style similarity weight vector to balance the importance of style similarity in the overall visual quality. The third deep neural network is an overall visual quality prediction network, which manipulates the content preservation weight vector to generate adaptive content preservation weight values, and manipulates the style similarity weight vector to generate adaptive style similarity weight values, and then fuses the content preservation quality score and the style similarity quality score into an overall visual quality score for the result image based on the adaptive content preservation weight values ​​and the adaptive style similarity weight values.

[0011] Step 2: Select N1 different pairs of style and content images; then use N2 style transfer algorithms to process each pair of style and content images, generating N2 different result images; then group each pair of style and content images with a corresponding result image into an image group, with N2 image groups corresponding to each pair of style and content images, and a total of N1×N2 image groups corresponding to N1 pairs of style and content images; next, use image cropping technology to process sub-blocks in the style image, content image, and result image of each image group, randomly selecting 25 sub-blocks in each image, with each sub-block having a size of 224×224, and the positions of the sub-blocks in the style image, content image, and result image of any image group are one-to-one; then, use image flipping technology with a certain probability of occurrence to process each sub-block in each style image, content image, and result image of each image group, transferring the style... Each sub-block in the image is flipped using an image flipping technique to obtain a 224×224 style image; each sub-block in the content image is flipped using an image flipping technique to obtain a 224×224 content image; and each sub-block in the result image is flipped using an image flipping technique to obtain a 224×224 result image. Finally, N1×N2×25 pairs of 224×224 style images, 224×224 content images, and 224×224 result images are used to form a training dataset. Each 224×224 style image in the training dataset is assigned the same subjective score as its original style image; each 224×224 content image is assigned the same subjective score as its original content image; and each 224×224 result image is assigned the same subjective score as its original result image. Where N1≥1 and N2≥1.

[0012] Step 3: For each pair of 224×224 style image, 224×224 content image, and 224×224 result image from the training dataset, use the 224×224 content image and 224×224 result image as the first RGB image and input them into the content-preserving quality prediction network for training. Before the end of each round of network training, calculate the loss function value of the content-preserving quality prediction network, denoted as Loss. CP , Then, the backpropagation algorithm is used to calculate the loss layer by layer through the Adam optimizer. CP For each neuron weight in each network layer, the partial derivatives are calculated, and the gradient descent algorithm is used to update the weights of each neuron in each network layer. For the pre-trained network layers, a 2×10⁻⁶ scale is applied. -5 The learning rate is set to 2×10 for untrained network layers. -4 The learning rate;

[0013] For each pair of 224×224 style images, 224×224 content images, and 224×224 result images in the training dataset, the 224×224 style image and the 224×224 result image are used as second RGB images and input into the style similarity quality prediction network for training. Before the end of each round of network training, the loss function value of the style similarity quality prediction network is calculated and denoted as Loss. SR , Then, the backpropagation algorithm is used to calculate the loss layer by layer through the Adam optimizer. SR For each neuron weight in each network layer, the partial derivatives are calculated, and the gradient descent algorithm is used to update the weights of each neuron in each network layer. For the pre-trained network layers, a 2×10⁻⁶ scale is applied. -5 The learning rate is set to 2×10 for untrained network layers. -4 The learning rate;

[0014] In the above, 1≤n≤N1×N2×25, and the symbol “||||1” indicates that the L1 norm of the element is being calculated. The subjective score represents the label value of the 224×224 content image in the nth pair of 224×224 style image, 224×224 content image, and 224×224 result image in the training dataset. This represents the content preservation quality score output by the content preservation quality prediction network after the nth pair of 224×224 style images, 224×224 content images, and 224×224 result images from the training dataset are input into the network. The label value, or subjective score, of the 224×224 style image in the nth pair of 224×224 style images, 224×224 content images, and 224×224 result images in the training dataset represents the subjective score. This represents the style similarity quality score output by the style similarity quality prediction network after the nth pair of 224×224 style images, 224×224 content images, and 224×224 result images in the training dataset are input into the style similarity quality prediction network.

[0015] Step 4: Repeat step 3 for a total of 15 rounds of network training. Every 5 rounds of network training, the learning rate is increased by 10 times. After 15 rounds of network training, the content-preserving quality prediction network training model and the style-similar quality prediction network training model are obtained and saved.

[0016] Step 5: For each pair of 224×224 style images, 224×224 content images, and 224×224 result images in the training dataset, use the 224×224 content image and 224×224 result image as the first RGB image and input them into the content preservation quality prediction network training model. The content preservation quality prediction network training model outputs the corresponding content preservation quality score and content preservation weight vector. For each pair of 224×224 style images, 224×224 content images, and 224×224 result images in the training dataset, use the 224×224 style image and 224×224 result image as the second RGB image and input them into the style similarity quality prediction network training model. The style similarity quality prediction network training model outputs the corresponding style similarity quality score and style similarity weight vector.

[0017] Step 6: For each pair of 224×224 style images, 224×224 content images, and 224×224 result images in the training dataset, use the 224×224 style image as the third RGB image. Then, input the third RGB image, the corresponding content preservation quality score and content preservation weight vector output by the content preservation quality prediction network training model, and the corresponding style similarity quality score and style similarity weight vector output by the style similarity quality prediction network training model into the overall visual quality prediction network for training. Before the end of one round of network training, calculate the loss function value of the overall visual quality prediction network, denoted as Loss. OV , Then, the backpropagation algorithm, through the Adam optimizer, is used to fine-tune the parameters in the quality prediction network training model and the style similarity quality prediction network training model, maintaining the parameters in both models, and calculating the loss layer by layer. OV For each neuron weight in each network layer, the partial derivatives are calculated, and the gradient descent algorithm is used to update the weights of each neuron in each network layer. For the pre-trained network layers, a 2×10⁻⁶ scale is applied. -5 The learning rate is set to 2×10 for untrained network layers. -4 The learning rate;

[0018] The above, The label value, or subjective score, represents the 224×224 result image among the nth pair of 224×224 style images, 224×224 content images, and 224×224 result images in the training dataset. This represents the overall visual quality score output by the overall visual quality prediction network after inputting the nth pair of 224×224 style images, 224×224 content images, and 224×224 result images from the training dataset, along with the content preservation quality score and content preservation weight vector output by the nth pair of 224×224 style images, 224×224 content images, and 224×224 result images from the training dataset, the nth pair of style similarity quality scores and style similarity weight vectors from the training model of the overall visual quality prediction network.

[0019] Step 7: Repeat step 6 for a total of 15 rounds of network training. The learning rate will be increased by 10 times every 5 rounds. After 15 rounds of network training, the overall visual quality prediction network training model will be obtained.

[0020] Step 8: Select any pair of style image, content image, and result image as the evaluation image to be tested; then input the content image and result image into the content preservation quality prediction network training model without fine-tuning parameters, and the content preservation quality prediction network training model outputs the content preservation quality score of the result image; input the style image and result image into the style similarity quality prediction network training model without fine-tuning parameters, and the style similarity quality prediction network training model outputs the style similarity quality score of the result image; input the content image and result image into the content preservation quality prediction network training model with fine-tuned parameters, input the style image and result image into the style similarity quality prediction network training model with fine-tuned parameters, input the style image into the overall visual quality prediction network training model, and the overall visual quality prediction network training model outputs the overall visual quality score of the result image; wherein, the size of the evaluation image to be tested is 224×224.

[0021] In step 1, the content-preserving quality prediction network includes a first feature extraction module, a first feature fusion module, and a first feature regression module; wherein,

[0022] The first feature extraction module mainly consists of the first ResNet50 network, the first to eighth convolutional layers, the first to fourth ReLU layers, the first max pooling layer, the first upsampling layer, and the first SoftMax layer. The input of the first ResNet50 network receives the R, G, and B channels of a first RGB image of size H×W. The first RGB image can be either the content image or the result image. The output of the first layer of the first ResNet50 network outputs the first feature of the first RGB image. The first ResNet50 network outputs a second feature map of the first RGB image, denoted as FC1. The output of the second layer of the first ResNet50 network is the second feature map of the first RGB image, denoted as FC2. The output of the third layer of the first ResNet50 network is the third feature map of the first RGB image, denoted as FC3. The input of the first convolutional layer receives FC1. FC1 is processed sequentially through the first convolutional layer, the first ReLU layer, and the first max-pooling layer. The feature map output by the first max-pooling layer is denoted as FC4. The input of the second convolutional layer is... The input of the third convolutional layer receives FC2, which is then processed by the second convolutional layer and the second ReLU layer. The feature map output by the second ReLU layer is denoted as FC5. The input of the third convolutional layer receives FC3, which is then processed by the third convolutional layer, the third ReLU layer, and the first upsampling layer. The feature map output by the first upsampling layer is denoted as FC6. FC4, FC5, and FC6 are concatenated along the channel dimension. The input of the fourth convolutional layer receives the result of this concatenation operation. The result is processed sequentially through the 4th convolutional layer and the 4th ReLU layer. The feature map output by the 4th ReLU layer is denoted as FC7. The input of the 7th convolutional layer receives FC7, and the feature map output by the 7th convolutional layer is denoted as FC8. A Whiten operation is performed on FC7. The inputs of the 5th and 6th convolutional layers receive the results of the Whiten operation, and the feature map output by the 5th convolutional layer is denoted as FC9. The feature map output by the 6th convolutional layer is denoted as FC9. 10 For FC9 and FC 10 The element-wise multiplication operation is performed, and the input of the first SoftMax layer receives FC9 and FC. 10 The result of the element-wise multiplication operation is denoted as the feature map output from the first SoftMax layer as FC. 11 For FC8 and FC 11 Perform element-wise multiplication, then multiply FC8 and FC8. 11 The result of the element-wise multiplication operation is added element-wise to FC7, and the result of the element-wise addition operation is denoted as FC. 12 The input of the 8th convolutional layer receives FC.12 When the first RGB image is the content image, the feature map output from the output of the 8th convolutional layer is denoted as... When the first RGB image is the result image, the feature map output from the output of the 8th convolutional layer is denoted as... Among them, the first to eighth convolutional layers are all 2D convolutional layers, and the input size of the first convolutional layer is [size missing]. Output size is The kernel size is 1×1, the stride is 1, and the auto-padding is 0. The input size of the second convolutional layer is... Output size is The kernel size is 1×1, the stride is 1, and the auto-padding is 0. The input size of the third convolutional layer is... Output size is The kernel size is 1×1, the stride is 1, and the auto-padding is 0. The input size of the 4th convolutional layer is... Output size is The kernel size is 1×1, the stride is 1, and the auto-padding is 0. The input size of the 5th convolutional layer is... Output size is The kernel size is 1×1, the stride is 1, and the auto-padding is 0. The input size of the 6th convolutional layer is... Output size is The kernel size is 1×1, the stride is 1, and the auto-padding is 0. The input size of the 7th convolutional layer is... Output size is The kernel size is 1×1, the stride is 1, and the auto-padding is 0. The input size of the 8th convolutional layer is... Output size is The convolution kernel size is 1×1, the stride is 1, and the auto-fill is 0. The input size of the first max pooling layer is... Output size is The convolution kernel size is 2×2, the stride is 2, and the auto-padding is 0. The input size of the first upsampling layer is... Output size is scale_factor = 2, the input size of the first SoftMax layer is Output size is Dim = -1, the input size of the first ResNet50 network is H×W×3, and the output size of the first layer of the first ResNet50 network is... The output size of the second layer is The output size of the third layer is

[0023] The first feature fusion module is to first... and Perform element-wise subtraction, then... and The result of element-wise subtraction and Perform a join operation along the channel dimension, and denote the result of the join operation as F. CP ;

[0024] The first feature regression module mainly consists of convolutional layers 9 through 12, ReLU layers 5 through 8, max pooling layer 2, max pooling layer 3, and fully connected layers 1 through 3. The inputs of the 9th and 11th convolutional layers respectively receive F... CP F CP The process sequentially passes through the 9th convolutional layer, the 5th ReLU layer, the 10th convolutional layer, the 6th ReLU layer, and the 2nd max-pooling layer. The feature map output from the 2nd max-pooling layer is denoted as... F CP The other path sequentially passes through the 11th convolutional layer, the 7th ReLU layer, the 12th convolutional layer, the 8th ReLU layer, and the 3rd max-pooling layer. The feature map output from the 3rd max-pooling layer is denoted as... right Perform a feature tiling operation and denote the resulting feature vector as V1. CP ;right Perform feature tiling operation, and denote the resulting feature vector as... The input of the first fully connected layer receives V1. CP V1 CP The content is processed sequentially through the first and second fully connected layers. The output value of the second fully connected layer is used as the content preservation quality score, denoted as Q. CP The input of the third fully connected layer receives... The vector output from the third fully connected layer is used as the content preservation weight vector, denoted as ω. CP Among them, the 9th to 12th convolutional layers are all 2D convolutional layers, and the input size of the 9th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, and the auto-padding is 0. The input size of the 10th convolutional layer is... Output size is The kernel size is 1×1, the stride is 1, and the auto-padding is 0. The input size of the 11th convolutional layer is... Output size is The kernel size is 1×1, the stride is 1, and the auto-padding is 0. The input size of the 12th convolutional layer is... Output size is The kernel size is 1×1, the stride is 1, and the auto-fill is 0. The input size of the second max pooling layer is... Output size is The kernel size is 7×7, the stride is 7, and the auto-fill is 0. The input size of the third max pooling layer is... Output size is The convolution kernel size is 7×7, the stride is 7, and the auto-fill is 0. The first fully connected layer has 1024 input neurons and 512 output neurons, the second fully connected layer has 512 input neurons and 1 output neuron, and the third fully connected layer has 1024 input neurons and 256 output neurons.

[0025] In step 1, the style similarity quality prediction network includes a second feature extraction module, a second feature fusion module, and a second feature regression module; wherein,

[0026] The second feature extraction module mainly consists of a second ResNet50 network, convolutional layers 13 through 16, and a second SoftMax layer. The input of the second ResNet50 network receives the R, G, and B channels of a second RGB image of size H×W. This second RGB image is either the result image or a style image. The output of the fourth layer of the second ResNet50 network outputs the feature map of the second RGB image, denoted as FS1. The inputs of the 15th, 13th, and 14th convolutional layers receive FS1 respectively. The feature map output by the 15th convolutional layer is denoted as FS2. The input of the 13th convolutional layer... The feature map output from the 14th convolutional layer is denoted as FS3, and the feature map output from the 14th convolutional layer is denoted as FS4. Channel-level element-wise multiplication is performed on FS3 and FS4. The input of the second SoftMax layer receives the result of this channel-level element-wise multiplication, and the feature map output from the second SoftMax layer is denoted as FS5. Element-wise multiplication is performed on FS2 and FS5, and then the result of this element-wise multiplication is added to FS1. The result of this addition is denoted as FS6. The input of the 16th convolutional layer receives FS6. When the second RGB image is the result image, the feature map output from the 16th convolutional layer is denoted as... When the second RGB image is a style image, the feature map output from the output of the 16th convolutional layer is denoted as... Among them, the 13th to 16th convolutional layers are all 2D convolutional layers, and the input size of the 13th to 16th convolutional layers is all... Output dimensions are all All convolutional kernels are 1×1, all strides are 1, and all auto-padding is 0. The input size of the second SoftMax layer is... Output size is Dim = -1, the input size of the second ResNet50 network is H×W×3, and the output size of the fourth layer of the second ResNet50 network is...

[0027] The second feature fusion module is to first... and Perform element-wise subtraction, then... and The result of element-wise subtraction and Perform a join operation along the channel dimension, and denote the result of the join operation as F. SR ;

[0028] The second feature regression module mainly consists of convolutional layers 17 to 21, ReLU layers 9 to 13, max pooling layer 4, max pooling layer 5, and fully connected layers 4 to 6. The inputs of the 17th and 20th convolutional layers respectively receive F... SR F SR The process sequentially passes through the 17th convolutional layer, the 9th ReLU layer, the 18th convolutional layer, the 10th ReLU layer, the 19th convolutional layer, the 11th ReLU layer, and the 4th max-pooling layer. The feature map output from the 4th max-pooling layer is denoted as... F SR The other path sequentially passes through the 20th convolutional layer, the 12th ReLU layer, the 21st convolutional layer, the 13th ReLU layer, and the 5th max-pooling layer. The feature map output from the 5th max-pooling layer is denoted as... right Perform feature tiling operation, and denote the resulting feature vector as... right Perform a feature tiling operation and denote the resulting feature vector as V1. SR The input of the fourth fully connected layer receives... The vector output from the fourth fully connected layer is used as the style similarity weight vector, denoted as ω. SR V1 SR The material is processed sequentially through the 5th and 6th fully connected layers. The output value of the 6th fully connected layer is used as the style similarity quality score, denoted as Q. SR Among them, the 17th to 21st convolutional layers are all 2D convolutional layers, and the input size of the 17th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, and the auto-padding is 0. The input size of the 18th convolutional layer is... Output size is The kernel size is 1×1, the stride is 1, and the auto-padding is 0. The input size of the 19th convolutional layer is... Output size is The kernel size is 1×1, the stride is 1, and the auto-padding is 0. The input size of the 20th convolutional layer is... Output size is The kernel size is 1×1, the stride is 1, and the auto-padding is 0. The input size of the 21st convolutional layer is... Output size is The kernel size is 1×1, the stride is 1, and the auto-fill is 0. The input size of the 4th max pooling layer is... Output size is The convolution kernel size is 2×2, the stride is 2, and the auto-fill is 0. The input size of the 5th max pooling layer is... Output size is The convolution kernel size is 14×14, the stride is 14, and the auto-fill is 0. The fourth fully connected layer has 980 input neurons and 256 output neurons, the fifth fully connected layer has 512 input neurons and 512 output neurons, and the sixth fully connected layer has 512 input neurons and 1 output neuron.

[0029] In step 1, the overall visual quality prediction network includes an adaptive weight estimation module and a score calculation module; wherein,

[0030] The adaptive weight estimation module mainly consists of the 3rd ResNet50 network, the 7th to 17th fully connected layers, the 1st to 6th adaptive average pooling layers, the 22nd to 25th convolutional layers, and the 14th to 18th RelU layers. The input of the 3rd ResNet50 network receives the R, G, and B channels of a third RGB image of size H×W, which is the style image. The feature map output from the second layer of the 3rd ResNet50 network is denoted as FO. The Gram matrix of FO is calculated, denoted as Gram, with a size of 256×256. The inputs of the 22nd, 1st, 23rd, 2nd, 24th, and 3rd adaptive average pooling layers, and the 25th convolutional layer are also included. The input terminals of the 22nd convolutional layer, the 4th adaptive average pooling layer, the 5th adaptive average pooling layer, and the 6th adaptive average pooling layer each receive Gram. The feature map output from the 22nd convolutional layer is denoted as A1, the feature map output from the 1st adaptive average pooling layer as B1, the feature map output from the 23rd convolutional layer as A2, the feature map output from the 2nd adaptive average pooling layer as B2, the feature map output from the 24th convolutional layer as A3, the feature map output from the 3rd adaptive average pooling layer as B3, the feature map output from the 25th convolutional layer as A4, the feature map output from the 4th adaptive average pooling layer as B4, the feature map output from the 5th adaptive average pooling layer as B5, and the feature map output from the 6th adaptive average pooling layer as B6. A1 is then re-denoted as... The size is 128×256; the input of the 12th fully connected layer receives B1, and the vector output by the output of the 12th fully connected layer is denoted as... A2 is reshaped into a new feature with dimensions of 64×128 using feature size reshaping technology, and denoted as... The input of the 13th fully connected layer receives B2, and the vector output from the output of the 13th fully connected layer is denoted as... The feature size A3 is reshaped into a new feature of size 32×64 using the feature size reshaping technique, and denoted as . The input of the 14th fully connected layer receives B3, and the vector output from the output of the 14th fully connected layer is denoted as... The A4 paper is reshaped into a new feature with a size of 16×32 using feature size reshaping technology, and denoted as... The input of the 15th fully connected layer receives B4, and the vector output from the output of the 15th fully connected layer is denoted as... The input of the 16th fully connected layer receives B5, and the vector output from the output of the 16th fully connected layer is denoted as... The input of the 17th fully connected layer receives B6, and the value output by the output of the 17th fully connected layer is denoted as... Will The eigenvalues ​​are used as the weight parameters of the 7th fully connected layer. The node values ​​are used as bias parameters for the 7th fully connected layer; The eigenvalues ​​are used as the weight parameters of the 8th fully connected layer. The node values ​​are used as bias parameters for the 8th fully connected layer; The eigenvalues ​​are used as the weight parameters of the 9th fully connected layer. The node values ​​are used as bias parameters for the 9th fully connected layer; The eigenvalues ​​are used as the weight parameters of the 10th fully connected layer. The node values ​​are used as bias parameters for the 10th fully connected layer; The node values ​​are used as weight parameters for the 11th fully connected layer. The node values ​​are used as bias parameters for the 11th fully connected layer; the input of the 7th fully connected layer receives ω. CP or ω SR ω CP or ω SR The data is processed sequentially through the 7th fully connected layer, the 14th RelU layer, the 8th fully connected layer, the 15th RelU layer, the 9th fully connected layer, the 16th RelU layer, the 10th fully connected layer, the 17th RelU layer, the 11th fully connected layer, and the 18th RelU layer. When the input of the 7th fully connected layer receives ω... CP The output of the 18th RelU layer retains the adaptive weight values, denoted as . When the input of the 7th fully connected layer receives ω SR The output style similarity adaptive weight value of the 18th RelU layer is denoted as . The input size of the third ResNet50 network is H×W×3, and the output size of the second layer of the third ResNet50 network is... Convolutional layers 22 through 25 are all 1D convolutional layers. The 22nd convolutional layer has an input size of 256×256, an output size of 128×256, a kernel size of 1×1, a stride of 1, and zero auto-padding. The 23rd convolutional layer has an input size of 256×256, an output size of 32×256, a kernel size of 1×1, a stride of 1, and zero auto-padding. The 24th convolutional layer has an input size of 256×256 and an output size of 8×256. The convolutional kernel size is 1×1, the stride is 1, and auto-padding is 0. The input size of the 25th convolutional layer is 256×256, the output size is 2×256, the kernel size is 1×1, the stride is 1, and auto-padding is 0. The 7th fully connected layer has 256 input neurons and 128 output neurons. The 8th fully connected layer has 128 input neurons and 64 output neurons. The 9th fully connected layer has 64 input neurons and 3 output neurons. 2. The 10th fully connected layer has 32 input neurons and 16 output neurons; the 11th fully connected layer has 16 input neurons and 1 output neuron; the 12th fully connected layer has 256 input neurons and 128 output neurons; the 13th fully connected layer has 256 input neurons and 64 output neurons; the 14th fully connected layer has 256 input neurons and 32 output neurons; the 15th fully connected layer has 256 input neurons and 16 output neurons; the 16th fully connected layer has 256 input neurons and 16 output neurons; and the 17th fully connected layer has 256 input neurons and 1 output neuron. All six adaptive average pooling layers (1 to 6) are 1D adaptive average pooling layers, with an input size of 256×256 and an output size of 1×256. The value of each coordinate point in each component is used as a feature value. Each value in each node is treated as a node value.

[0031] The fraction calculation module is based on Q CP , and Q SR , The overall visual quality score of the calculated image is denoted as .

[0032] The process of obtaining the Gram matrix of FO is as follows: The size is... The FO re-aggregates into a size of The two-dimensional feature map is denoted as FO′; then Gram = (FO′)(FO′) is calculated. TAmong them, (FO′) T This represents the transpose of FO′.

[0033] Compared with the prior art, the advantages of the present invention are as follows:

[0034] 1) The method of this invention can obtain reliable quality factor scores (e.g., content preservation quality factor and style similarity quality factor). Based on the powerful feature extraction capabilities of convolutional neural networks (deep neural networks), this method extracts features from content-style image pairs (content image and style image) and their corresponding style transfer images (result image). Then, this method proposes a feature fusion module that allows joint regression of the features extracted from the style transfer image (result image) and the original content (or style) image to obtain different reliable quality factor scores. These reliable quality factor scores not only provide an objective evaluation of the content preservation and style similarity of the transferred image, but also facilitate the training of a reliable overall visual quality prediction network model, thereby effectively improving the consistency between the objective evaluation results of content preservation, style similarity, and overall visual quality and the results of human subjective perception.

[0035] 2) The method of this invention can balance the impact of different quality factors on the overall quality. Based on the observation that style type affects human judgment of the importance of different factors, this invention does not use fixed weights to regress the final quality. Instead, it follows the human top-down perception process and designs a new adaptive weight estimation strategy. The key principle of this strategy is to treat style type information as a pooling rule and adaptively manipulate the weight vector to transform it into importance weights. The method of this invention has been tested on a large scale on existing style transfer image quality assessment databases, further demonstrating the effectiveness of the method.

[0036] 3) The method of this invention can obtain adaptive weights. The method of this invention designs an adaptive weight estimation module, which can adaptively generate weight parameters according to the input style type features, and convert the weight vector into weights in a style type-aware manner. In this way, the method of this invention becomes more adaptive, allowing for more flexible and accurate manipulation of quality factor weights when facing complex content style image pairs, thereby obtaining a more reliable overall visual quality score. Attached Figure Description

[0037] Figure 1 This is a block diagram illustrating the overall implementation of the method of the present invention;

[0038] Figure 2 This is a block diagram showing the composition of the content preservation quality prediction network constructed in the method of this invention;

[0039] Figure 3This is a block diagram of the style similarity quality prediction network constructed in the method of the present invention;

[0040] Figure 4 This is a block diagram showing the composition of the style adaptive weight estimation module in the overall visual quality prediction network constructed in the method of this invention. Detailed Implementation

[0041] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0042] The present invention proposes a style transfer image quality assessment method, the overall implementation framework of which is as follows: Figure 1 As shown, it includes the following steps:

[0043] Step 1: Construct three deep neural networks: The first deep neural network is a content preservation quality prediction network, which extracts features from both the content image and the result image, then fuses the features of the content image and the result image, and obtains a content preservation quality score to measure the structural content similarity between the content image and the result image, and a content preservation weight vector to balance the importance of content preservation in the overall visual quality. The second deep neural network is a style similarity quality prediction network, which extracts features from both the style image and the result image, then fuses the features of the style image and the result image, and obtains a style similarity quality score to measure the style style similarity between the style image and the result image, and a style similarity weight vector to balance the importance of style similarity in the overall visual quality. The third deep neural network is an overall visual quality prediction network, which manipulates the content preservation weight vector to generate adaptive content preservation weight values, and manipulates the style similarity weight vector to generate adaptive style similarity weight values, and then fuses the content preservation quality score and the style similarity quality score into an overall visual quality score for the result image based on the adaptive content preservation weight values ​​and the adaptive style similarity weight values.

[0044] Step 2: Select N1 different pairs of style images and content images; then use N2 existing style transfer algorithms to process each pair of style images and content images, generating N2 different result images (i.e., style transfer images); then group each pair of style images and content images with a corresponding result image into an image group, with N2 image groups corresponding to each pair of style images and content images, and a total of N1×N2 image groups corresponding to N1 pairs of style images and content images; next, use existing image cropping techniques to extract sub-blocks from the style image, content image, and result image in each image group, randomly selecting 25 sub-blocks of 224×224 size from each image, with the positions of the sub-blocks in the style image, content image, and result image of any image group corresponding one-to-one; then, use existing image flipping techniques with a certain probability of occurrence to process each sub-block in each style image, content image, and result image of each image group, and use the result of the image flipping technique on each sub-block of the style image as the result of the image flipping technique. A 224×224 style image is used as the basis for a 224×224 content image, and a 224×224 result image is created by flipping each sub-block of the content image. Finally, N1×N2×25 pairs of 224×224 style images, 224×224 content images, and 224×224 result images form a training dataset. Each 224×224 style image, each 224×224 content image, and each 224×224 result image in the training dataset are assigned the same subjective score as their original style image, the same subjective score as their original content image, and the same subjective score as their original result image. Here, N1≥1 (e.g., N1=120), N2≥1 (e.g., N2=8), and eight different style transfer algorithms are defined as WCT(Y.Li,C.Fang,J.Yang,Z.Wang,X.Lu,and M.-H. Yang, “Universal style transfer via feature transforms,” Advances in neural information processing systems, vol.30, 2017. (Research progress on universal style transfer based on feature transforms and neural information processing systems) and LST (X.Li, S.Liu, J.Kautz, and M.-H. Yang).Yang, “Learning linear transformations for fast image and video style transfer,” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 3809–3817. SEMST (C. Chen, “Structure-emphasized multimodal style transfer,” Master-Tokyo Institute of Technology: Tokyo, Japan, 2020). AdaIN (X. Huang and S. Belongie, “Arbitrary style transfer in real-time with adaptive instance normalization,” in Proceedings of the IEEE international conference on computer... Vision, 2017, pp. 1501–1510. (Real-time Arbitrary Style Transfer with Adaptive Instance Normalization, IEEE International Conference on Computer Vision)); AAMS (Y. Yao, J. Ren, X. Xie, W. Liu, Y.-J. Liu, and J. Wang, “Attention-aware multi-stroke style transfer,” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 1467–1475. (Style Transfer Based on Multi-Stroke Attention, IEEE Conference on Computer Vision and Pattern Recognition)); MANet (Y. Deng, F. Tang, W. Dong, W. Sun, F. Huang, and C. Xu, “Arbitrary style transfer via multi-adaptation network,” in Proceedings of the 28th National Congress of the Communist Party of China). thACM International Conference on Multimedia, 2020, pp. 2719–2727. (Arbitrary Style Transfer Based on Multimodal Networks, ACM International Conference on Multimedia) SANet (DYPark and KHLee, “Arbitrary style transfer with style-attentional networks,” in proceedings of the IEEE / CVF conference on computer vision and pattern recognition, 2019, pp. 5880–5888. (Arbitrary style transfer networks with style attention, IEEE Conference on Computer Vision and Pattern Recognition) MST (Y. Zhang, C. Fang, Y. Wang, Z. Wang, Z. Lin, Y. Fu, and J. Yang, “Multimodal style transfer via graph cuts,” in Proceedings of the IEEE / CVF International Conference on Computer Vision and Pattern Recognition) Vision, 2019, pp.5943–5951. (Multimodal style transfer based on image segmentation, IEEE Conference on Computer Vision and Pattern Recognition)). The positions of sub-blocks in different image groups can correspond (be the same) or not correspond (be different). In this case, the style image, content image and result image are directly selected from the style transfer image quality evaluation database established by the Multimedia Laboratory of Ningbo University. The size of the style image, content image and result image is 512×512.

[0045] Step 3: For each pair of 224×224 style image, 224×224 content image, and 224×224 result image from the training dataset, use the 224×224 content image and 224×224 result image as the first RGB image and input them into the content-preserving quality prediction network for training. Before the end of each round of network training, calculate the loss function value of the content-preserving quality prediction network, denoted as Loss. CP , Then, the backpropagation algorithm is used to calculate the loss layer by layer through the Adam optimizer. CP For each neuron weight in each network layer, the partial derivative is calculated, and the gradient descent algorithm is used to update the weights of each neuron in each network layer. For pre-trained network layers (such as the first ResNet50 network), a 2×10⁻⁶ gradient is applied. -5The learning rate is set to 2×10 for untrained network layers. -4 The learning rate.

[0046] For each pair of 224×224 style images, 224×224 content images, and 224×224 result images in the training dataset, the 224×224 style image and the 224×224 result image are used as second RGB images and input into the style similarity quality prediction network for training. Before the end of each round of network training, the loss function value of the style similarity quality prediction network is calculated and denoted as Loss. SR , Then, the backpropagation algorithm is used to calculate the loss layer by layer through the Adam optimizer. SR For each neuron weight in each network layer, the partial derivative is calculated, and the gradient descent algorithm is used to update the weights of each neuron in each network layer. For pre-trained network layers (such as the second ResNet50 network), a 2×10⁻⁶ gradient is applied. -5 The learning rate is set to 2×10 for untrained network layers. -4 The learning rate.

[0047] In the above, 1≤n≤N1×N2×25, and the symbol “||||1” indicates that the L1 norm of the element is being calculated. The subjective score represents the label value of the 224×224 content image in the nth pair of 224×224 style image, 224×224 content image, and 224×224 result image in the training dataset. This represents the content preservation quality score output by the content preservation quality prediction network after the nth pair of 224×224 style images, 224×224 content images, and 224×224 result images from the training dataset are input into the network. The label value, or subjective score, of the 224×224 style image in the nth pair of 224×224 style images, 224×224 content images, and 224×224 result images in the training dataset represents the subjective score. This represents the style similarity quality score output by the style similarity quality prediction network after the nth pair of 224×224 style images, 224×224 content images, and 224×224 result images in the training dataset are input into the style similarity quality prediction network.

[0048] Step 4: Repeat Step 3 for a total of 15 rounds of network training. Every 5 rounds, the learning rate is increased by a factor of 10. After 15 rounds of training, the content-preserving quality prediction network training model and the style-similar quality prediction network training model are obtained and saved. In practice, two copies of the content-preserving quality prediction network training model and the style-similar quality prediction network training model are saved: one copy is used for subsequent feasibility and effectiveness testing, and the other copy is used for fine-tuning parameters to train the overall visual quality prediction network.

[0049] Step 5: For each pair of 224×224 style images, 224×224 content images, and 224×224 result images in the training dataset, use the 224×224 content image and 224×224 result image as the first RGB image and input them into the content preservation quality prediction network training model. The content preservation quality prediction network training model outputs the corresponding content preservation quality score and content preservation weight vector. For each pair of 224×224 style images, 224×224 content images, and 224×224 result images in the training dataset, use the 224×224 style image and 224×224 result image as the second RGB image and input them into the style similarity quality prediction network training model. The style similarity quality prediction network training model outputs the corresponding style similarity quality score and style similarity weight vector.

[0050] Step 6: For each pair of 224×224 style images, 224×224 content images, and 224×224 result images in the training dataset, use the 224×224 style image as the third RGB image. Then, input the third RGB image, the corresponding content preservation quality score and content preservation weight vector output by the content preservation quality prediction network training model, and the corresponding style similarity quality score and style similarity weight vector output by the style similarity quality prediction network training model into the overall visual quality prediction network for training. Before the end of one round of network training, calculate the loss function value of the overall visual quality prediction network, denoted as Loss. OV , Then, the backpropagation algorithm, through the Adam optimizer, is used to fine-tune the parameters in the quality prediction network training model and the style similarity quality prediction network training model, maintaining the parameters in both models, and calculating the loss layer by layer. OV For each neuron weight in each network layer, the partial derivative is calculated, and the gradient descent algorithm is used to update the weights of each neuron in each network layer. For pre-trained network layers (such as the third ResNet50 network), a 2×10⁻⁶ gradient is applied. -5 The learning rate is set to 2×10 for untrained network layers. -4The learning rate is used. To build a more reliable overall visual quality prediction network training model, the Adam optimizer, in addition to updating the weight parameters of each layer in the overall visual quality prediction network, also fine-tunes the parameters of the saved content preservation quality prediction network training model and style similarity quality prediction network training model through backpropagation. The content preservation quality prediction network training model and style similarity quality prediction network training model with fine-tuned parameters are no longer used for performance testing. The purpose of fine-tuning is to train a more reliable overall visual quality prediction network training model. Therefore, the content preservation quality prediction network training model and style similarity quality prediction network training model need to be backed up in advance. The content preservation quality prediction network training model and style similarity quality prediction network training model without fine-tuned parameters should be used for content preservation and style similarity performance testing.

[0051] The above, The label value, or subjective score, represents the 224×224 result image among the nth pair of 224×224 style images, 224×224 content images, and 224×224 result images in the training dataset. This represents the overall visual quality score output by the overall visual quality prediction network after inputting the nth pair of 224×224 style images, 224×224 content images, and 224×224 result images from the training dataset, along with the content preservation quality score and content preservation weight vector output by the nth pair of 224×224 style images, 224×224 content images, and 224×224 result images from the training dataset, the nth pair of style similarity quality scores and style similarity weight vectors from the training model of the overall visual quality prediction network.

[0052] Step 7: Repeat step 6 for a total of 15 rounds of network training. Every 5 rounds of network training, the learning rate is increased by 10 times. After 15 rounds of network training, the overall visual quality prediction network training model is obtained.

[0053] Step 8: Select any pair of style image, content image, and result image as the evaluation image to be tested; then input the content image and result image into the content preservation quality prediction network training model without fine-tuning parameters, and the content preservation quality prediction network training model outputs the content preservation quality score of the result image; input the style image and result image into the style similarity quality prediction network training model without fine-tuning parameters, and the style similarity quality prediction network training model outputs the style similarity quality score of the result image; input the content image and result image into the content preservation quality prediction network training model with fine-tuned parameters, input the style image and result image into the style similarity quality prediction network training model with fine-tuned parameters, input the style image into the overall visual quality prediction network training model, and the overall visual quality prediction network training model outputs the overall visual quality score of the result image; wherein, the size of the evaluation image to be tested is 224×224.

[0054] In this embodiment, in step 1, as follows Figure 2 As shown, the content-preserving quality prediction network includes a first feature extraction module, a first feature fusion module, and a first feature regression module; wherein,

[0055] The first feature extraction module mainly consists of a first ResNet50 network, convolutional layers C1 to C8, ReLU layers R1 to R4, a first max pooling layer M1, a first upsampling layer U1, and a first SoftMax layer S1. The input of the first ResNet50 network receives the R, G, and B channels of a first RGB image of size H×W. This first RGB image can be either the content image or the result image. The output of the first layer of the first ResNet50 network outputs the first feature map of the first RGB image. The first ResNet50 network outputs a second feature map of the first RGB image, denoted as FC1. The output of the second layer of the first ResNet50 network is the third feature map of the first RGB image, denoted as FC2. The output of the third layer of the first ResNet50 network is the third feature map of the first RGB image, denoted as FC3. The input of the first convolutional layer C1 receives FC1. FC1 is processed sequentially through the first convolutional layer C1, the first ReLU layer R1, and the first max pooling layer M1. The feature map output by the first max pooling layer M1 is denoted as FC4. The input of the second convolutional layer C2 receives FC2. FC2 is processed sequentially through... The second convolutional layer C2 and the second ReLU layer R2 process the data, and the feature map output from the second ReLU layer R2 is denoted as FC5. The input of the third convolutional layer C3 receives FC3, which is then processed sequentially through the third convolutional layer C3, the third ReLU layer R3, and the first upsampling layer U1. The feature map output from the first upsampling layer U1 is denoted as FC6. A concatenation operation is performed on FC4, FC5, and FC6 along the channel dimension. The input of the fourth convolutional layer C4 receives the result of this concatenation operation. The result is processed sequentially through the 4th convolutional layer C4 and the 4th ReLU layer R4. The feature map output from the 4th ReLU layer R4 is denoted as FC7. The input of the 7th convolutional layer C7 receives FC7, and the feature map output from the 7th convolutional layer C7 is denoted as FC8. A Whiten operation is performed on FC7. The inputs of the 5th convolutional layer C5 and the 6th convolutional layer C6 receive the results of the Whiten operation, and the feature map output from the 5th convolutional layer C5 is denoted as FC9, and the feature map output from the 6th convolutional layer C6 is denoted as FC8. 10 For FC9 and FC 10 The element-wise multiplication operation is performed, and the input of the first SoftMax layer S1 receives FC9 and FC. 10 The result of the element-wise multiplication operation is denoted as the feature map output from the output of the first SoftMax layer S1 as FC. 11 For FC8 and FC 11 Perform element-wise multiplication, then multiply FC8 and FC8.11 The result of the element-wise multiplication operation is added element-wise to FC7, and the result of the element-wise addition operation is denoted as FC. 12 The input of the 8th convolutional layer C8 receives FC. 12 When the first RGB image is the content image, the feature map output from the output of the 8th convolutional layer C8 is denoted as... When the first RGB image is the result image, the feature map output from the output of the 8th convolutional layer C8 is denoted as... Among them, the first convolutional layer C1 to the eighth convolutional layer C8 are all 2D convolutional layers, and the input size of the first convolutional layer C1 is... Output size is The convolutional kernel size is 1×1, the stride is 1, and the auto-padding is 0. The input size of the second convolutional layer C2 is... Output size is The kernel size is 1×1, the stride is 1, and the auto-padding is 0. The input size of the third convolutional layer C3 is... Output size is The convolutional kernel size is 1×1, the stride is 1, and the auto-padding is 0. The input size of the 4th convolutional layer C4 is... Output size is The kernel size is 1×1, the stride is 1, and the auto-padding is 0. The input size of the 5th convolutional layer C5 is... Output size is The kernel size is 1×1, the stride is 1, and the auto-padding is 0. The input size of the 6th convolutional layer C6 is... Output size is The kernel size is 1×1, the stride is 1, and the auto-padding is 0. The input size of the 7th convolutional layer C7 is... Output size is The kernel size is 1×1, the stride is 1, and the auto-padding is 0. The input size of the 8th convolutional layer C8 is... Output size is The convolution kernel size is 1×1, the stride is 1, and the auto-fill is 0. The input size of the first max-pooling layer M1 is... Output size is The convolution kernel size is 2×2, the stride is 2, and the auto-fill is 0. The input size of the first upsampling layer U1 is... Output size is scale_factor = 2, the input size of the first SoftMax layer S1 is Output size is Dim = -1, the input size of the first ResNet50 network is H×W×3, and the output size of the first layer of the first ResNet50 network is... The output size of the second layer is The output size of the third layer is The ResNet50 network is an existing network whose structure is publicly available, as documented in the paper K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778. The ResNet50 network has five layers: layers 1, 2, 3, 4, and 5. In the content-preserving quality prediction network, only layers 1, 2, and 3 of the ResNet50 network were used. Concatenation, element-wise multiplication, and element-wise addition operations are all existing techniques. The Whiten operation is an existing image detexturing operation, described in the paper Y. Li, C. Fang, J. Yang, Z. Wang, X. Lu, and M.-H. Yang, “Universal style transfer via feature transforms,” Advances in neural information processing. It has been described in systems, vol.30, 2017 (General Style Transfer Based on Feature Transformation).

[0056] The first feature fusion module is to first... and Perform element-wise subtraction, then... and The result of element-wise subtraction and Perform a join operation along the channel dimension, and denote the result of the join operation as F. CP ; where element subtraction is an existing operation, here, element subtraction is... Subtracting the elements in The corresponding element in.

[0057] The first feature regression module mainly consists of the 9th convolutional layer C9 to the 12th convolutional layer C12, the 5th ReLU layer R5 to the 8th ReLU layer R8, the 2nd max pooling layer M2, the 3rd max pooling layer M3, and the 1st fully connected layer L1 to the 3rd fully connected layer L3; the input terminals of the 9th convolutional layer C9 and the 11th convolutional layer C11 respectively receive F CP F CP The process sequentially passes through the 9th convolutional layer C9, the 5th ReLU layer R5, the 10th convolutional layer C10, the 6th ReLU layer R6, and the 2nd max pooling layer M2. The feature map output from the 2nd max pooling layer M2 is denoted as F1. CP F CP The other path sequentially passes through the 11th convolutional layer C11, the 7th ReLU layer R7, the 12th convolutional layer C12, the 8th ReLU layer R8, and the 3rd max pooling layer M3. The feature map output from the 3rd max pooling layer M3 is denoted as... For F1 CP Perform a feature tiling operation and denote the resulting feature vector as V1. CP ;right Perform feature tiling operation, and denote the resulting feature vector as... The input of the first fully connected layer L1 receives V1. CP V1 CP The content is processed sequentially through the first fully connected layer L1 and the second fully connected layer L2. The output value of the second fully connected layer L2 is used as the content preservation quality score, denoted as Q. CP The input of the third fully connected layer L3 receives... The vector output from the third fully connected layer L3 is used as the content preservation weight vector, denoted as ω. CP Among them, the 9th convolutional layer C9 to the 12th convolutional layer C12 are all 2D convolutional layers, and the input size of the 9th convolutional layer C9 is [missing information]. Output size is The kernel size is 1×1, the stride is 1, and the auto-padding is 0. The input size of the 10th convolutional layer C10 is... Output size is The kernel size is 1×1, the stride is 1, and the auto-padding is 0. The input size of the 11th convolutional layer C11 is... Output size is The kernel size is 1×1, the stride is 1, and the auto-padding is 0. The input size of the 12th convolutional layer C12 is... Output size is The convolution kernel size is 1×1, the stride is 1, and the auto-fill is 0. The input size of the second max-pooling layer M2 is... Output size is The convolution kernel size is 7×7, the stride is 7, and the auto-fill is 0. The input size of the third max-pooling layer M3 is... Output size is The convolution kernel size is 7×7, the stride is 7, and the auto-fill is 0. The first fully connected layer L1 has 1024 input neurons and 512 output neurons, the second fully connected layer L2 has 512 input neurons and 1 output neuron, and the third fully connected layer L3 has 1024 input neurons and 256 output neurons. The feature tiling operation is an existing operation method.

[0058] In this embodiment, in step 1, as follows Figure 3 As shown, the style similarity quality prediction network includes a second feature extraction module, a second feature fusion module, and a second feature regression module; among which,

[0059] The second feature extraction module mainly consists of a second ResNet50 network, 13th to 16th convolutional layers C13 to C16, and a second SoftMax layer S2. The input of the second ResNet50 network receives the R, G, and B channels of a second RGB image of size H×W. This second RGB image is either a result image or a style image, and it is the same result image as the first RGB image. This style image and the content image of the first RGB image form a content-style image pair. The output of the fourth layer of the second ResNet50 network outputs the feature map of the second RGB image, denoted as FS1. The inputs of the 15th convolutional layer C15, the 13th convolutional layer C13, and the 14th convolutional layer C14 respectively receive FS1, and the 15th convolutional layer... The feature map output from layer C15 is denoted as FS2, the feature map output from the 13th convolutional layer C13 is denoted as FS3, and the feature map output from the 14th convolutional layer C14 is denoted as FS4. Channel-level element-wise multiplication is performed on FS3 and FS4. The input of the second SoftMax layer S2 receives the result of the channel-level element-wise multiplication of FS3 and FS4, and the feature map output from the second SoftMax layer S2 is denoted as FS5. Element-wise multiplication is performed on FS2 and FS5, and then the result of the element-wise multiplication of FS2 and FS5 is added to FS1. The result of the element-wise addition is denoted as FS6. The input of the 16th convolutional layer C16 receives FS6. When the second RGB image is the result image, the feature map output from the 16th convolutional layer C16 is denoted as... When the second RGB image is a style image, the feature map output from the output of the 16th convolutional layer C16 is denoted as... Among them, the 13th convolutional layer C13 to the 16th convolutional layer C16 are all 2D convolutional layers, and the input size of the 13th convolutional layer C13 to the 16th convolutional layer C16 is all... Output dimensions are all All convolutional kernels are 1×1, all strides are 1, and all auto-padding is 0. The input size of the second SoftMax layer S2 is... Output size is Dim = -1, the input size of the second ResNet50 network is H×W×3, and the output size of the fourth layer of the second ResNet50 network is... In the style similarity quality prediction network, only the fourth layer of the ResNet50 network was used, and the channel-level element-wise multiplication operation adopted the existing operation method.

[0060] The second feature fusion module is to first... and Perform element-wise subtraction, then... and The result of element-wise subtraction and Perform a join operation along the channel dimension, and denote the result of the join operation as F. SR Here, the element subtraction operation is... Subtracting the elements in The corresponding element in.

[0061] The second feature regression module mainly consists of the 17th convolutional layer C17 to the 21st convolutional layer C21, the 9th ReLU layer R9 to the 13th ReLU layer R13, the 4th max pooling layer M4, the 5th max pooling layer M5, and the 4th fully connected layer L4 to the 6th fully connected layer L6; the input terminals of the 17th convolutional layer C17 and the 20th convolutional layer C20 respectively receive F SR F SR The process sequentially passes through the 17th convolutional layer C17, the 9th ReLU layer R9, the 18th convolutional layer C18, the 10th ReLU layer R10, the 19th convolutional layer C19, the 11th ReLU layer R11, and the 4th max pooling layer M4. The feature map output from the 4th max pooling layer M4 is denoted as... F SR The other path sequentially passes through the 20th convolutional layer C20, the 12th ReLU layer R12, the 21st convolutional layer C21, the 13th ReLU layer R13, and the 5th max pooling layer M5. The feature map output from the 5th max pooling layer M5 is denoted as F1. SR ;right Perform feature tiling operation, and denote the resulting feature vector as... For F1SR Perform a feature tiling operation and denote the resulting feature vector as V1. SR The input of the fourth fully connected layer L4 receives... The vector output from the fourth fully connected layer L4 is used as the style similarity weight vector, denoted as ω. SR V1 SR The process proceeds sequentially through the 5th fully connected layer L5 and the 6th fully connected layer L6. The output value of the 6th fully connected layer L6 is taken as the style similarity quality score, denoted as Q. SR Among them, the 17th convolutional layer C17 to the 21st convolutional layer C21 are all 2D convolutional layers, and the input size of the 17th convolutional layer C17 is... Output size is The kernel size is 1×1, the stride is 1, and the auto-padding is 0. The input size of the 18th convolutional layer C18 is... Output size is The kernel size is 1×1, the stride is 1, and the auto-padding is 0. The input size of the 19th convolutional layer C19 is... Output size is The kernel size is 1×1, the stride is 1, and the auto-padding is 0. The input size of the 20th convolutional layer C20 is... Output size is The kernel size is 1×1, the stride is 1, and the auto-padding is 0. The input size of the 21st convolutional layer C21 is... Output size is The convolution kernel size is 1×1, the stride is 1, and the auto-fill is 0. The input size of the fourth max-pooling layer M4 is... Output size is The convolution kernel size is 2×2, the stride is 2, and the auto-fill is 0. The input size of the 5th max-pooling layer M5 is... Output size is The convolution kernel size is 14×14, the stride is 14, and the auto-fill is 0. The fourth fully connected layer L4 has 980 input neurons and 256 output neurons. The fifth fully connected layer L5 has 512 input neurons and 512 output neurons. The sixth fully connected layer L6 has 512 input neurons and 1 output neuron. The feature tiling operation is an existing operation method.

[0062] In this embodiment, in step 1, the overall visual quality prediction network includes an adaptive weight estimation module and a score calculation module; wherein,

[0063] like Figure 4As shown, the adaptive weight estimation module mainly consists of the third ResNet50 network, the seventh fully connected layer L7 to the seventeenth fully connected layer L17, the first adaptive average pooling layer P1 to the sixth adaptive average pooling layer P6, the twenty-second convolutional layer C22 to the twenty-fifth convolutional layer C25, and the fourteenth RelU layer R14 to the eighteenth RelU layer R18. The input of the third ResNet50 network receives the R, G, and B channels of a third RGB image of size H×W, which is a style image. The style image is the same style image as the second RGB image. The feature map output from the second layer of the third ResNet50 network is denoted as FO. The Gram matrix of FO is calculated and denoted as Gram, with a size of 256×256. The inputs of the 22nd convolutional layer C22, the 1st adaptive average pooling layer P1, the 23rd convolutional layer C23, the 2nd adaptive average pooling layer P2, the 24th convolutional layer C24, and the 3rd adaptive average pooling layer P1 are... The inputs of the adaptive average pooling layer P3, the 25th convolutional layer C25, the 4th adaptive average pooling layer P4, the 5th adaptive average pooling layer P5, and the 6th adaptive average pooling layer P6 receive Gram respectively. The feature map output from the 22nd convolutional layer C22 is denoted as A1, the feature map output from the 1st adaptive average pooling layer P1 is denoted as B1, and the feature map output from the 23rd convolutional layer C23 is denoted as A2. The input from the 2nd adaptive average pooling layer P2... The feature map output from the first convolutional layer is denoted as B2. The feature map output from the 24th convolutional layer C24 is denoted as A3. The feature map output from the 3rd adaptive average pooling layer P3 is denoted as B3. The feature map output from the 25th convolutional layer C25 is denoted as A4. The feature map output from the 4th adaptive average pooling layer P4 is denoted as B4. The feature map output from the 5th adaptive average pooling layer P5 is denoted as B5. The feature map output from the 6th adaptive average pooling layer P6 is denoted as B6. A1 is re-denoted as... The size is 128×256; the input of the 12th fully connected layer L12 receives B1, and the vector output from the output of the 12th fully connected layer L12 is denoted as... Using existing feature size reshaping techniques, A2 is reshaped into a new feature with dimensions of 64×128, and denoted as... The input of the 13th fully connected layer L13 receives B2, and the vector output from the output of the 13th fully connected layer L13 is denoted as... The feature size A3 is reshaped into a new feature of size 32×64 using the feature size reshaping technique, and denoted as . The input of the 14th fully connected layer L14 receives B3, and the vector output from the output of the 14th fully connected layer L14 is denoted as... The A4 paper is reshaped into a new feature with a size of 16×32 using feature size reshaping technology, and denoted as... The input of the 15th fully connected layer L15 receives B4, and the vector output from the output of the 15th fully connected layer L15 is denoted as... The input of the 16th fully connected layer L16 receives B5, and the vector output from the output of the 16th fully connected layer L16 is denoted as... The input of the 17th fully connected layer L17 receives B6, and the value output from the output of the 17th fully connected layer L17 is denoted as... Using the F.Conv function in PyTorch The eigenvalues ​​are used as the weight parameters of the 7th fully connected layer L7. The node values ​​are used as bias parameters for the 7th fully connected layer L7; The eigenvalues ​​are used as the weight parameters of the 8th fully connected layer L8. The node values ​​are used as bias parameters for the 8th fully connected layer, L8; The eigenvalues ​​are used as the weight parameters of the 9th fully connected layer L9. The node values ​​are used as bias parameters for the 9th fully connected layer, L9; The eigenvalues ​​are used as the weight parameters of the 10th fully connected layer L10. The node values ​​are used as bias parameters for the 10th fully connected layer L10; The node values ​​are used as weight parameters for the 11th fully connected layer L11. The node values ​​are used as bias parameters for the 11th fully connected layer L11; the input of the 7th fully connected layer L7 receives ω. CP or ω SR ω CP or ω SR The data is processed sequentially through the 7th fully connected layer L7, the 14th RelU layer R14, the 8th fully connected layer L8, the 15th RelU layer R15, the 9th fully connected layer L9, the 16th RelU layer R16, the 10th fully connected layer L10, the 17th RelU layer R17, the 11th fully connected layer L11, and the 18th RelU layer R18. When the input of the 7th fully connected layer L7 receives ω... CP The output of the 18th RelU layer (R18) retains the adaptive weight values, denoted as... When the input of the 7th fully connected layer L7 receives ω SR The output style similarity adaptive weight value of the 18th RelU layer R18 is denoted as . The input size of the third ResNet50 network is H×W×3, and the output size of the second layer of the third ResNet50 network is... Convolutional layers C22 through C25 are all 1D convolutional layers. The input size of convolutional layer C22 is 256×256, the output size is 128×256, the kernel size is 1×1, the stride is 1, and auto-padding is 0. The input size of convolutional layer C23 is 256×256, the output size is 32×256, the kernel size is 1×1, the stride is 1, and auto-padding is 0. The input size of convolutional layer C24 is 256×256, the output size is 8×256, and the kernel size is... The kernel size is 1×1, stride is 1, and auto-padding is 0. The 25th convolutional layer C25 has an input size of 256×256, an output size of 2×256, a kernel size of 1×1, stride of 1, and auto-padding of 0. The 7th fully connected layer L7 has 256 input neurons and 128 output neurons. The 8th fully connected layer L8 has 128 input neurons and 64 output neurons. The 9th fully connected layer L9 has 64 input neurons and 32 output neurons. The 10th fully connected layer L10 has... The 11th fully connected layer L11 has 16 input neurons and 1 output neuron. The 12th fully connected layer L12 has 256 input neurons and 128 output neurons. The 13th fully connected layer L13 has 256 input neurons and 64 output neurons. The 14th fully connected layer L14 has 256 input neurons and 32 output neurons. The 15th fully connected layer L15 has 256 input neurons and 16 output neurons. The 16th fully connected layer L16 has 256 input neurons and 16 output neurons. The 17th fully connected layer L17 has 256 input neurons and 1 output neuron. The first adaptive average pooling layer P1 to the sixth adaptive average pooling layer P6 are all 1D adaptive average pooling layers. The input size of the first adaptive average pooling layer P1 to the sixth adaptive average pooling layer P6 is 256×256, and the output size is 1×256. Calculating the Gram matrix of an image is a prior art technique. The value of each coordinate point in each component is used as a feature value. Each value in each node is treated as a node value.

[0064] The fraction calculation module is based on Q CP , and Q SR , The overall visual quality score of the calculated image is denoted as .

[0065] The above process for obtaining the Gram matrix of FO is as follows: using existing technology, the size of the Gram matrix is... The FO re-aggregates into a size of The two-dimensional feature map is denoted as FO′; then Gram = (FO′)(FO′) is calculated. T Among them, (FO′) T This represents the transpose of FO′.

[0066] To further demonstrate the feasibility and effectiveness of the method of the present invention, experiments were conducted on the method of the present invention.

[0067] The method of this invention was tested on a style transfer image quality evaluation database constructed by the Multimedia Laboratory of Ningbo University. This database contains 150 pairs of content images and style images. Eight representative style transfer algorithms (such as the eight style transfer algorithms disclosed in step 2) were then used to process the 150 pairs of content images and style images, ultimately generating 1200 result images. Forty-five experts and students from the fields of art and image processing subjectively evaluated each result image in three aspects: content preservation, style similarity, and overall visual quality.

[0068] The method of this invention employs an evaluation protocol widely used in existing deep learning-based quality assessment research. Specifically, 20% of the image pairs (30 pairs of content images and style images) from the aforementioned database were randomly selected for testing, resulting in a test dataset of 30 × 8 × 25 = 6000 pairs of 224 × 224 style images, 224 × 224 content images, and 224 × 224 result images. The remaining 80% of the image pairs (120 pairs of content images and style images) were used for network training, resulting in a training dataset of 120 × 8 × 25 = 24000 pairs of 224 × 224 style images, 224 × 224 content images, and 224 × 224 result images. The entire training and testing process was repeated five times to ensure that the test dataset was distributed across the entire database. For each process, the method of this invention used four commonly used metrics to evaluate the model's performance: Spearman Rank Correlation Coefficient (SRCC), Kendall Rank Correlation Coefficient (KRCC), Pearson Linear Correlation Coefficient (PLCC), and Root Mean Square Error (RMSE). PLCC and RMSE measure prediction accuracy, while SRCC and KRCC measure prediction monotonicity. Since label values ​​are only meaningful within the same group, these metrics are calculated separately for each group from the same source image. Finally, the average of five test datasets is reported as the final performance metric to eliminate sample bias. Superior performance will have higher SRCC, PLCC, and KRCC values, ranging from [-1, 1], with larger absolute values ​​indicating better performance. Superior performance will also have a lower RMSE value (minimum value of 0). Specifically, before calculating PLCC, the method of this invention fits a five-parameter function to compensate for nonlinearity: Where f(x) is the subjective score, x is the quality prediction score, and τ i (i = 1, 2, ..., 5) are the fitting parameters, e is the natural cardinality, e = 2.71...

[0069] Table 1 shows the SRCC, KRCC, PLCC, and RMSE performance results of the method of the present invention in terms of content retention.

[0070] Table 1. Performance results of the method of the present invention in terms of content retention.

[0071] SRCC KRCC PLCC RMSE Method of the present invention 0.8374 0.7309 0.9142 1.3097

[0072] Table 2 shows the performance results of the method of the present invention in terms of SRCC, KRCC, PLCC and RMSE in terms of style similarity.

[0073] Table 2 shows the performance results of the method of the present invention in terms of style similarity.

[0074] SRCC KRCC PLCC RMSE Method of the present invention 0.6579 0.5389 0.6943 1.0295

[0075] Table 3 shows the SRCC, KRCC, PLCC, and RMSE performance results of the method of the present invention in overall vision.

[0076] Table 3 shows the performance results of the method of the present invention in overall vision.

[0077] SRCC KRCC PLCC RMSE Method of the present invention 0.6657 0.5382 0.7402 0.7815

[0078] As can be seen from the results in Tables 1, 2 and 3, the method of the present invention has achieved good results in the existing style transfer image quality evaluation database. This fully demonstrates the effectiveness and feasibility of the method of the present invention, and also proves that the method of the present invention is highly consistent with human subjective cognition.

Claims

1. A method for evaluating image quality through style transfer, characterized in that... Includes the following steps: Step 1: Construct three deep neural networks: a content preservation quality prediction network, a style similarity quality prediction network, and an overall visual quality prediction network. The content preservation quality prediction network is used to obtain the content preservation quality score and the content preservation weight vector. The style similarity quality prediction network is used to obtain the style similarity quality score and the style similarity weight vector. The overall visual quality prediction network is used to manipulate the content preservation weight vector to generate the content preservation adaptive weight value and manipulate the style similarity weight vector to generate the style similarity adaptive weight value. Then, the content preservation quality score and the style similarity quality score are fused into the overall visual quality score based on the content preservation adaptive weight value and the style similarity adaptive weight value. Step 2: Select N1 different pairs of style images and content images; process each pair of style images and content images using N2 style transfer algorithms to generate N2 different result images, where N1≥1 and N2≥1; group each pair of style images and content images with a corresponding result image into an image group, resulting in N1×N2 image groups; use image cropping techniques to extract sub-blocks from each image in each image group, randomly selecting 25 sub-blocks from each image, with each sub-block having a size of 224×224; use image flipping techniques to process the sub-blocks to obtain style images, content images, and result images with a size of 224×224, thereby constructing a training dataset, and assigning each image the same subjective score as the original image it belongs to; Step 3: Take each pair of content images and result images in the training dataset as the first RGB image and input them into the content-preserving quality prediction network for training; Each pair of style images and result images in the training dataset is used as a second RGB image and input into the style similarity quality prediction network for training. Step 4: Repeat step 3 for 15 rounds of training to obtain the content preservation quality prediction network training model and the style similarity quality prediction network training model; Step 5: Input the first RGB image into the content preservation quality prediction network training model, and output the corresponding content preservation quality score and content preservation weight vector; input the second RGB image into the style similarity quality prediction network training model, and output the corresponding style similarity quality score and style similarity weight vector. Step 6: Use the style image in the training dataset as the third RGB image, and input it together with the content preservation quality score and the content preservation weight vector, the style similarity quality score and the style similarity weight vector into the overall visual quality prediction network for training. During the training process, fine-tune the parameters in the content preservation quality prediction network training model and the style similarity quality prediction network training model obtained in Step 4. Step 7: Repeat step 6 for 15 rounds of training to obtain the overall visual quality prediction network training model; Step 8: Select the style image, content image, and result image to be tested; input the content image and result image into the fine-tuned content preservation quality prediction network training model to obtain the content preservation quality score and content preservation weight vector; input the style image and result image into the fine-tuned style similarity quality prediction network training model to obtain the style similarity quality score and style similarity weight vector; input the style image, content preservation quality score and content preservation weight vector, style similarity quality score and style similarity weight vector into the overall visual quality prediction network training model to obtain the overall visual quality score of the result image.

2. The style transfer image quality assessment method according to claim 1, characterized in that... In step 1, the content-preserving quality prediction network includes a first feature extraction module, a first feature fusion module, and a first feature regression module; wherein, The first feature extraction module mainly consists of the first ResNet50 network, the first to eighth convolutional layers, the first to fourth ReLU layers, the first max pooling layer, the first upsampling layer, and the first SoftMax layer; the input of the first ResNet50 network receives a frame of size... The first RGB image contains the R, G, and B channels. This first RGB image can be either the content image or the result image. The first feature map of the first RGB image is output from the first layer of the first ResNet50 network and denoted as... The output of the second layer of the first ResNet50 network is the second feature map of the first RGB image, denoted as... The output of the third layer of the first ResNet50 network is the third feature map of the first RGB image, denoted as... The input of the first convolutional layer receives... , The feature map output from the first max pooling layer is denoted as follows: [Description of feature map]. The process involves sequentially passing through the first convolutional layer, the first ReLU layer, and the first max pooling layer. The input of the second convolutional layer receives... , The feature map output from the second ReLU layer is denoted as follows: After passing through the second convolutional layer and the second ReLU layer sequentially, the feature map is denoted as... The input of the third convolutional layer receives... , The feature map output from the first upsampling layer is denoted as follows: After passing through the third convolutional layer, the third ReLU layer, and the first upsampling layer, the feature map is denoted as... ;right , and A concatenation operation is performed along the channel dimension. The input of the fourth convolutional layer receives the result of the concatenation operation. The result of the concatenation operation is then processed sequentially through the fourth convolutional layer and the fourth ReLU layer. The feature map output by the fourth ReLU layer is denoted as... The input of the 7th convolutional layer receives... The feature map output from the 7th convolutional layer is denoted as... ;right The Whiten operation is performed, with the inputs of the 5th and 6th convolutional layers receiving the results of the Whiten operation respectively. The feature map output by the 5th convolutional layer is denoted as... The feature map output from the output of the 6th convolutional layer is denoted as... ;right and Element-wise multiplication is performed, and the input of the first SoftMax layer receives... and The result of the element-wise multiplication operation is denoted as the feature map output from the first SoftMax layer. ;right and Perform element-wise multiplication, then... and The result of element-wise multiplication and Perform element-wise addition and denote the result as . The input of the 8th convolutional layer receives... When the first RGB image is the content image, the feature map output from the output of the 8th convolutional layer is denoted as... When the first RGB image is the result image, the feature map output from the output of the 8th convolutional layer is denoted as... Among them, the first to eighth convolutional layers are all 2D convolutional layers, and the input size of the first convolutional layer is... Output size is The kernel size is The stride is 1, auto-fill is 0, and the input size of the second convolutional layer is... Output size is The kernel size is The stride is 1, auto-fill is 0, and the input size of the third convolutional layer is... Output size is The kernel size is The stride is 1, auto-fill is 0, and the input size of the 4th convolutional layer is... Output size is The kernel size is The stride is 1, auto-fill is 0, and the input size of the 5th convolutional layer is... Output size is The kernel size is The stride is 1, auto-fill is 0, and the input size of the 6th convolutional layer is [value missing]. Output size is The kernel size is The stride is 1, auto-fill is 0, and the input size of the 7th convolutional layer is [value missing]. Output size is The kernel size is The stride is 1, auto-fill is 0, and the input size of the 8th convolutional layer is [value missing]. Output size is The kernel size is The sliding step size is 1, the auto-fill is 0, and the input size of the first max pooling layer is... Output size is The kernel size is The sliding step size is 2, the autofill is 0, and the input size of the first upsampling layer is... Output size is With scale_factor = 2, the input size of the first SoftMax layer is... Output size is Dim = -1, the input size of the first ResNet50 network is The output size of the first layer of the first ResNet50 network is The output size of the second layer is The output size of the third layer is ; The first feature fusion module is to first... and Perform element-wise subtraction, then... and The result of element-wise subtraction and Perform a join operation along the channel dimension, and denote the result of the join operation as... ; The first feature regression module mainly consists of convolutional layers 9 through 12, ReLU layers 5 through 8, max pooling layer 2, max pooling layer 3, and fully connected layers 1 through 3. The inputs of the 9th and 11th convolutional layers respectively receive... , The process sequentially passes through the 9th convolutional layer, the 5th ReLU layer, the 10th convolutional layer, the 6th ReLU layer, and the 2nd max-pooling layer. The feature map output from the 2nd max-pooling layer is denoted as... , The other path sequentially passes through the 11th convolutional layer, the 7th ReLU layer, the 12th convolutional layer, the 8th ReLU layer, and the 3rd max-pooling layer. The feature map output from the 3rd max-pooling layer is denoted as... ;right Perform feature tiling operation, and denote the resulting feature vector as... ;right Perform feature tiling operation, and denote the resulting feature vector as... The input of the first fully connected layer receives... , The content is processed sequentially through the first and second fully connected layers. The output value of the second fully connected layer is used as the content preservation quality score, denoted as... The input of the third fully connected layer receives... The vector output from the third fully connected layer is used as the content preservation weight vector, denoted as... Among them, the 9th to 12th convolutional layers are all 2D convolutional layers, and the input size of the 9th convolutional layer is [missing information]. Output size is The kernel size is The stride is 1, auto-fill is 0, and the input size of the 10th convolutional layer is... Output size is The kernel size is The stride is 1, auto-fill is 0, and the input size of the 11th convolutional layer is... Output size is The kernel size is The stride is 1, auto-fill is 0, and the input size of the 12th convolutional layer is... Output size is The kernel size is The sliding step size is 1, the auto-fill is 0, and the input size of the second max pooling layer is... Output size is The kernel size is The sliding step size is 7, the auto-fill is 0, and the input size of the third max pooling layer is... Output size is The kernel size is The sliding step size is 7, the auto-fill is 0, the number of input neurons in the first fully connected layer is 1024, the number of output neurons is 512, the number of input neurons in the second fully connected layer is 512, the number of output neurons is 1, and the number of input neurons in the third fully connected layer is 1024, the number of output neurons is 256.

3. The style transfer image quality assessment method according to claim 2, characterized in that... In step 1, the style similarity quality prediction network includes a second feature extraction module, a second feature fusion module, and a second feature regression module; wherein, The second feature extraction module mainly consists of the second ResNet50 network, the 13th to 16th convolutional layers, and the second SoftMax layer; the input of the second ResNet50 network receives a pattern of size... The second RGB image contains the R, G, and B channels. This second RGB image is the result image or style image. The output of the fourth layer of the second ResNet50 network is the feature map of the second RGB image, denoted as... The input terminals of the 15th, 13th, and 14th convolutional layers respectively receive... The feature map output from the 15th convolutional layer is denoted as... The feature map output from the 13th convolutional layer is denoted as... The feature map output from the 14th convolutional layer is denoted as... ;right and Perform channel-level element-wise multiplication; the input of the second SoftMax layer receives... and The result of the channel-level element-wise multiplication operation is denoted as the feature map output from the output of the second SoftMax layer. ;right and Perform element-wise multiplication, then... and The result of element-wise multiplication and Perform element-wise addition and denote the result as . The input of the 16th convolutional layer receives... When the second RGB image is the result image, the feature map output from the output of the 16th convolutional layer is denoted as... When the second RGB image is a style image, the feature map output from the output of the 16th convolutional layer is denoted as... Among them, the 13th to 16th convolutional layers are all 2D convolutional layers, and the input size of the 13th to 16th convolutional layers is... The output dimensions are all The kernel size is 1. The sliding step size is 1, and the autofill size is 0. The input size of the second SoftMax layer is... Output size is Dim = -1, the input size of the second ResNet50 network is The output size of the fourth layer of the second ResNet50 network is ; The second feature fusion module is to first... and Perform element-wise subtraction, then... and The result of element-wise subtraction and Perform a join operation along the channel dimension, and denote the result of the join operation as... ; The second feature regression module mainly consists of convolutional layers 17 to 21, ReLU layers 9 to 13, max pooling layer 4, max pooling layer 5, and fully connected layers 4 to 6. The inputs of the 17th and 20th convolutional layers respectively receive... , The process sequentially passes through the 17th convolutional layer, the 9th ReLU layer, the 18th convolutional layer, the 10th ReLU layer, the 19th convolutional layer, the 11th ReLU layer, and the 4th max-pooling layer. The feature map output from the 4th max-pooling layer is denoted as... , The other path sequentially passes through the 20th convolutional layer, the 12th ReLU layer, the 21st convolutional layer, the 13th ReLU layer, and the 5th max-pooling layer. The feature map output from the 5th max-pooling layer is denoted as... ;right Perform feature tiling operation, and denote the resulting feature vector as... ;right Perform feature tiling operation, and denote the resulting feature vector as... The input of the fourth fully connected layer receives... The vector output from the fourth fully connected layer is used as the style similarity weight vector, denoted as... ; The process passes through the 5th and 6th fully connected layers sequentially. The output value of the 6th fully connected layer is used as the style similarity quality score, denoted as... Among them, the 17th to 21st convolutional layers are all 2D convolutional layers, and the input size of the 17th convolutional layer is [missing information]. Output size is The kernel size is The sliding stride is 1, auto-fill is 0, and the input size of the 18th convolutional layer is [value missing]. Output size is The kernel size is The sliding stride is 1, auto-fill is 0, and the input size of the 19th convolutional layer is [value missing]. Output size is The kernel size is The stride is 1, auto-fill is 0, and the input size of the 20th convolutional layer is [value missing]. Output size is The kernel size is The stride is 1, auto-fill is 0, and the input size of the 21st convolutional layer is... Output size is The kernel size is The sliding step size is 1, the auto-fill is 0, and the input size of the 4th max pooling layer is... Output size is The kernel size is The sliding step size is 2, the auto-fill is 0, and the input size of the 5th max pooling layer is... Output size is The kernel size is The sliding step size is 14, the auto-fill is 0, the number of input neurons in the 4th fully connected layer is 980, the number of output neurons is 256, the number of input neurons in the 5th fully connected layer is 512, the number of output neurons is 512, and the number of input neurons in the 6th fully connected layer is 512 and the number of output neurons is 1.

4. The style transfer image quality assessment method according to claim 3, characterized in that... In step 1, the overall visual quality prediction network includes an adaptive weight estimation module and a score calculation module; wherein, The adaptive weight estimation module mainly consists of the 3rd ResNet50 network, the 7th to 17th fully connected layers, the 1st to 6th adaptive average pooling layers, the 22nd to 25th convolutional layers, and the 14th to 18th RelU layers; the input of the 3rd ResNet50 network receives a frame of size... The third RGB image contains the R, G, and B channels. This third RGB image is the style image. The feature map output from the second layer of the third ResNet50 network is denoted as... ;calculate The Gram matrix, denoted as , The size is The inputs of the 22nd convolutional layer, the 1st adaptive average pooling layer, the 23rd convolutional layer, the 2nd adaptive average pooling layer, the 24th convolutional layer, the 3rd adaptive average pooling layer, the 25th convolutional layer, the 4th adaptive average pooling layer, the 5th adaptive average pooling layer, and the 6th adaptive average pooling layer respectively receive... The feature map output from the 22nd convolutional layer is denoted as... The feature map output from the first adaptive average pooling layer is denoted as... The feature map output from the 23rd convolutional layer is denoted as... The feature map output from the second adaptive average pooling layer is denoted as... The feature map output from the 24th convolutional layer is denoted as... The feature map output from the third adaptive average pooling layer is denoted as... The feature map output from the 25th convolutional layer is denoted as... The feature map output from the fourth adaptive average pooling layer is denoted as... The feature map output from the fifth adaptive average pooling layer is denoted as... The feature map output from the output of the 6th adaptive average pooling layer is denoted as... ;Will Re-recorded , The size is The input of the 12th fully connected layer receives... Let the vector output from the 12th fully connected layer be denoted as... ;Utilizing feature size reshaping technology to Remodeled to a size of The new features are denoted as The input of the 13th fully connected layer receives... Let the vector output from the 13th fully connected layer be denoted as... ;Utilizing feature size reshaping technology to Remodeled to a size of The new features are denoted as The input of the 14th fully connected layer receives... Let the vector output from the output of the 14th fully connected layer be denoted as... ;Utilizing feature size reshaping technology to Remodeled to a size of The new features are denoted as The input of the 15th fully connected layer receives... Let the vector output from the output of the 15th fully connected layer be denoted as... The input of the 16th fully connected layer receives... Let the vector output from the output of the 16th fully connected layer be denoted as... The input of the 17th fully connected layer receives... The value output by the 17th fully connected layer is denoted as... ;Will The eigenvalues ​​are used as the weight parameters of the 7th fully connected layer. The node values ​​are used as bias parameters for the 7th fully connected layer; The eigenvalues ​​are used as the weight parameters of the 8th fully connected layer. The node values ​​are used as bias parameters for the 8th fully connected layer; The eigenvalues ​​are used as the weight parameters of the 9th fully connected layer. The node values ​​are used as bias parameters for the 9th fully connected layer; The eigenvalues ​​are used as the weight parameters of the 10th fully connected layer. The node values ​​are used as bias parameters for the 10th fully connected layer; The node values ​​are used as weight parameters for the 11th fully connected layer. The node values ​​are used as bias parameters for the 11th fully connected layer; the input of the 7th fully connected layer receives... or , or The data is processed sequentially through the 7th fully connected layer, the 14th RelU layer, the 8th fully connected layer, the 15th RelU layer, the 9th fully connected layer, the 16th RelU layer, the 10th fully connected layer, the 17th RelU layer, the 11th fully connected layer, and the 18th RelU layer. When the input of the 7th fully connected layer receives... The output of the 18th RelU layer retains the adaptive weight values, denoted as . When the input of the 7th fully connected layer receives The output style similarity adaptive weight value of the 18th RelU layer is denoted as . The input size of the third ResNet50 network is... The output size of the second layer of the third ResNet50 network is Convolutional layers 22 through 25 are all 1D convolutional layers, with the input size of the 22nd convolutional layer being [missing value]. Output size is The kernel size is The stride is 1, auto-fill is 0, and the input size of the 23rd convolutional layer is... Output size is The kernel size is The sliding stride is 1, auto-fill is 0, and the input size of the 24th convolutional layer is [value missing]. Output size is The kernel size is The stride is 1, auto-fill is 0, and the input size of the 25th convolutional layer is [value missing]. Output size is The kernel size is The sliding step size is 1, and auto-fill is 0. The number of input neurons in the 7th fully connected layer is 256, and the number of output neurons is 128. The number of input neurons in the 8th fully connected layer is 128, and the number of output neurons is 64. The number of input neurons in the 9th fully connected layer is 64, and the number of output neurons is 32. The number of input neurons in the 10th fully connected layer is 32, and the number of output neurons is 16. The number of input neurons in the 11th fully connected layer is 16, and the number of output neurons is 1. The number of input neurons in the 12th fully connected layer is 256, and the number of output neurons is 128. The number of input neurons in the 13th fully connected layer is... The first fully connected layer has 256 input neurons and 64 output neurons. The 14th fully connected layer has 256 input neurons and 32 output neurons. The 15th fully connected layer has 256 input neurons and 16 output neurons. The 16th fully connected layer has 256 input neurons and 16 output neurons. The 17th fully connected layer has 256 input neurons and 1 output neuron. The first to sixth adaptive average pooling layers are all 1D adaptive average pooling layers. The input size of the first to sixth adaptive average pooling layers is [missing information]. The output dimensions are all , , , , The value of each coordinate point in each component is used as a feature value. , , , , , Each value in each node is treated as a node value. The fraction calculation module is based on , and , The overall visual quality score of the calculated image is denoted as . , .

5. The style transfer image quality assessment method according to claim 4, characterized in that... The aforementioned Gram matrix The process of obtaining it is as follows: The size is... of Repolymerized into a size of The two-dimensional feature map, denoted as Then calculate ;in, express The transpose of .

Images