An objective evaluation method for style transfer image quality based on sparse representation

Abstract

The present invention discloses an objective evaluation method for style transfer image quality based on sparse representation, which obtains structural images of different scales corresponding to each content image, and performs dictionary training operations on a set consisting of all sub-blocks in all structural images belonging to the same scale in an unsupervised learning manner to construct a multi-scale content dictionary; obtains a three-dimensional feature map of five convolutional layers corresponding to each artistic style image, and then obtains a style feature vector based on a Gram matrix of each three-dimensional feature map, and performs dictionary training operations on a set consisting of style feature vectors of all three-dimensional feature maps belonging to the same layer in an unsupervised learning manner to construct a multi-scale style dictionary; during testing, a content sparse coefficient matrix and a style sparse coefficient matrix are obtained according to the multi-scale content dictionary and the multi-scale style dictionary, and then an objective evaluation prediction value of the quality of the style transfer image is calculated; the advantage is that the correlation between the objective evaluation result and the subjective perception of the human eye can be improved.

Description

Technical Field

[0001] The present invention relates to an image quality evaluation method, and in particular to an objective evaluation method for style migration image quality based on sparse representation. Background Art

[0002] With the rapid development of image editing technology and artistic creative industries, image style transfer technology has received more and more attention and application, and has become a current research hotspot. Image style transfer technology is a technology that applies the style of a style image to another content image while keeping the structure of the content image unchanged. Although the most advanced methods have shown great success in image style transfer, it is difficult for these methods to distinguish their relative advantages, and the image quality prediction of style transfer has been a long-standing problem. However, there is currently a lack of effective objective evaluation methods for the quality of style transferred images. Therefore, it is of great significance to establish an effective objective evaluation model for style transfer image quality.

[0003] Since there are many factors that affect the quality of style transfer images, such as the structural fidelity of the content image, the transfer of the style image, the overall visual situation, and the visual fatigue of the observer, how to effectively evaluate the quality of style transfer images is a difficult problem that needs to be solved urgently. At present, existing methods focus on limited factors of style transfer quality, but lack fine-grained quality factors, have certain limitations, and cannot effectively match the aesthetic perception of human observers in practice. Sparse representation is to decompose the signal on a known set of functions, and strive to approximate the original signal with as few basis functions as possible in the transform domain. Current research mainly focuses on dictionary construction and sparse decomposition. A key issue of sparse representation is how to effectively construct a dictionary to represent the essential features of the image. Therefore, how to construct a dictionary that can reflect the essential features of the style transfer image and how to effectively use the dictionary to perform comprehensive quality estimation of the style transfer image in multiple dimensions are issues that need to be studied and solved in the process of objectively evaluating the quality of style transfer images. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide an objective evaluation method for style transfer image quality based on sparse representation, which can effectively improve the correlation between objective evaluation results and subjective perception of the human eye, and has low computational complexity and does not require the subjective evaluation value of each evaluation image to be known in advance.

[0005] The technical solution adopted by the present invention to solve the above technical problems is: a method for objectively evaluating the quality of style transfer images based on sparse representation, characterized by comprising two processes: a training phase and a testing phase, wherein the training phase comprises a multi-scale content dictionary training phase and a multi-scale style dictionary training phase;

[0006] The specific steps of the multi-scale content dictionary training phase are as follows:

[0007] Step 1_1: Select N c The original undistorted natural images constitute the content image training set, denoted as {IC i |1≤i≤N c}; where N c ≥1, IC i Indicates {IC i |1≤i≤N c}, i is a positive integer, 1≤i≤N c ,{IC i |1≤i≤N c} Each content image in has a width of WC and a height of HC;

[0008] Step 1_2: Use multi-scale Gaussian difference filtering technology to {IC i |1≤i≤N c} implements multi-scale structure separation for each content image in {IC i |1≤i≤N c}, and the structure images of different scales corresponding to each content image in {IC i |1≤i≤N c The set of structural images of different scales corresponding to all content images in} is recorded as Wherein, z and o are both positive integers, 1≤z≤Z, 1≤o≤O, Z represents the total number of groups in the Gaussian pyramid used in the multi-scale Gaussian difference filtering technology, O represents the total number of layers in each group in the Gaussian pyramid used in the multi-scale Gaussian difference filtering technology, Indicates IC i The corresponding structural image of the oth layer in the zth group, The width is WC and the height is HC;

[0009] Step 1_3: Each structural image in non-overlapping sub-blocks of size 8×8; then N of the same scale c All sub-blocks in the structured image form a set. N of the oth layer in the zth group c The set of all sub-blocks in the structured image is recorded as Then use the online dictionary learning method to N of the same scale c The dictionary training operation is performed on the set of all sub-blocks in the structured image, and the N of the same scale c The content dictionary table corresponding to the set of all sub-blocks in the structured image will be The corresponding content dictionary obtained by performing the dictionary training operation is denoted as CD z,o Finally, we get The corresponding multi-scale content dictionary is denoted as {CD z,o |1≤z≤Z,1≤o≤O}; where WC and HC are both divisible by 8, express N of the oth layer in the zth group c is a column vector consisting of the pixel values ​​of all pixels in the t-th sub-block of all sub-blocks in the structured image. The dimension is 64×1, CD z,o The dimension is 64×K, where K represents the number of atoms in the set dictionary, K≥1;

[0010] The specific steps of the multi-scale style dictionary training phase are as follows:

[0011] Step 2_1: Select N s The original artistic style images constitute the artistic style image training set, denoted as {IS j |1≤j≤N s}; where N s ≥1, IS j Indicates {IS j |1≤j≤N s}, j is a positive integer, 1≤j≤N s , I.S. j The width is WS j And the height is HS j ;

[0012] Step 2_2: Use the image deep structure and texture similarity measurement criteria to measure {IS j |1≤j≤N s} to generate each artistic style image in {IS j |1≤j≤N s The three-dimensional feature map of the five convolutional layers corresponding to each artistic style image in {IS j |1≤j≤N s The set of three-dimensional feature maps of the five convolutional layers corresponding to all artistic style images in} is recorded as Recalculate The style feature vector based on the Gram matrix of each three-dimensional feature map in The style feature vector based on the Gram matrix is ​​recorded as Among them, 1≤γ≤5, Indicates IS j The corresponding 3D feature map of the γth convolutional layer, The dimension is C γ represents the number of channels of the 3D feature map generated by the γth convolutional layer in the image deep structure and texture similarity metric, Expressing support for IS j The height of the 3D feature map generated by the γth convolutional layer in the image deep structure and texture similarity metric is Height, Expressing support for IS j The width of the 3D feature map generated by the γth convolutional layer in the image deep structure and texture similarity metric, i.e. The width of The dimension is C γ ×1;

[0013] Step 2_3: N belongs to the same convolutional layer s The style feature vectors based on the Gram matrix of the three-dimensional feature map form a set. N belongs to the γth convolutional layer s The set of style feature vectors based on the Gram matrix of a three-dimensional feature map is recorded as Then use the online dictionary learning method to N belongs to the same convolutional layer s The dictionary training operation is performed on the set of style feature vectors based on the Gram matrix of the three-dimensional feature map, and the N belongs to the same convolutional layer s The style dictionary table corresponding to the set of style feature vectors based on the Gram matrix of the three-dimensional feature map will be The corresponding style dictionary obtained by performing the dictionary training operation is denoted as SD γ Finally, we get The corresponding multi-scale style dictionary is denoted as {SD γ |1≤γ≤5}; where SD γ The dimension is C γ ×B γ , B γ represents the number of atoms in the dictionary set for the γth convolutional layer, B γ ≥1;

[0014] The specific steps of the testing phase are as follows:

[0015] Step 3_1: For any tested style transfer image, record it as R test ; and R test The corresponding original content image is denoted as IC org , R test The corresponding original artistic style image is denoted as IS org ; Among them, R test 、IC org The width is WC and the height is HC, IS org The width is WS org And the height is HS org ;

[0016] Step 3_2: Follow the process of step 1_2 to obtain R in the same way test Corresponding structural images and ICs at different scales org Corresponding structural images of different scales, R test The corresponding set of structural images of different scales is recorded as IC org The corresponding set of structural images of different scales is recorded as in, Represents R test The corresponding structural image of the oth layer in the zth group, Indicates IC org The corresponding structural image of the oth layer in the zth group, The width is WC and the height is HC;

[0017] Step 3_3: Each structural image in non-overlapping sub-blocks of size 8×8; Each structural image in non-overlapping sub-blocks of size 8×8; then All sub-blocks in each structural image in form a set. The set of all sub-blocks in is recorded as and will All sub-blocks in each structural image in form a set. The set of all sub-blocks in is recorded as Then according to {CD z,o |1≤z≤Z,1≤o≤O}, get The content sparse coefficient matrix of each element in the set composed of all sub-blocks in the structure image of the corresponding scale in is The sparse coefficient matrix of the content is recorded as Similarly, according to {CDz,o |1≤z≤Z,1≤o≤O}, get The content sparse coefficient matrix of each element in the set composed of all sub-blocks in the structure image of the corresponding scale in is The sparse coefficient matrix of the content is recorded as in, express A column vector consisting of the pixel values ​​of all pixels in the t'th sub-block, The dimension is 64×1, express A column vector consisting of the pixel values ​​of all pixels in the t'th sub-block, The dimension is 64×1, The dimensions of are all K×1;

[0018] Step 3_4: According to The content sparse coefficient matrix of each element in the set composed of all sub-blocks in each structural image in The content sparse coefficient matrix of each element in the set of all sub-blocks in each structural image in R is calculated. test The objective evaluation prediction value of content retention is denoted as Q content ;

[0019] Step 3_5: Follow the process of step 2_2 to obtain R in the same way test The corresponding three-dimensional feature map and IS of the five convolutional layers org The corresponding three-dimensional feature map of the five convolutional layers is R test The corresponding set of three-dimensional feature maps of the five convolutional layers is recorded as IS org The corresponding set of three-dimensional feature maps of the five convolutional layers is recorded as Then Each 3D feature map in is reaggregated into a dimension of C γ ×(HC γ ×WC γ )’s two-dimensional feature map, The re-aggregated two-dimensional feature map is recorded as Likewise, Each 3D feature map in is reaggregated into a dimension of The two-dimensional feature map of The re-aggregated two-dimensional feature map is recorded as Recalculate Each three-dimensional feature map in is re-aggregated into the Gram matrix of the two-dimensional feature map, The Gram matrix of Similarly, calculation Each three-dimensional feature map in is re-aggregated into the Gram matrix of the two-dimensional feature map, The Gram matrix of Among them, 1≤γ≤5, Represents R test The corresponding 3D feature map of the γth convolutional layer, Indicates IS org The corresponding 3D feature map of the γth convolutional layer, The dimension is C γ ×HC γ ×WC γ , The dimension is C γ represents the number of channels of the 3D feature map generated by the γth convolutional layer in the image deep structure and texture similarity metric, HC γ Indicates R test The height of the 3D feature map generated by the γth convolutional layer in the image deep structure and texture similarity metric, WC γ Indicates R test The width of the 3D feature map generated by the γth convolutional layer in the image deep structure and texture similarity metric, Expressing support for IS org The height of the 3D feature map generated by the γth convolutional layer in the image deep structure and texture similarity metric, Expressing support for IS org The width of the 3D feature map generated by the γth convolutional layer in the image deep structure and texture similarity metric, The dimensions are all C γ ×C γ , the superscript “T” indicates the transpose of a vector or matrix;

[0020] Step 3_6: According to {SD γ |1≤γ≤5}, get The style sparse coefficient matrix of the Gram matrix of the two-dimensional feature map of the three-dimensional feature map of the corresponding convolution layer in is re-aggregated, and the The style sparse coefficient matrix is ​​recorded as Similarly, according to {SD γ |1≤γ≤5}, get The style sparse coefficient matrix of the Gram matrix of the two-dimensional feature map of the three-dimensional feature map of the corresponding convolution layer in is re-aggregated, and the The style sparse coefficient matrix is ​​recorded as in, The dimensions are all B γ ×C γ , B γ ≥1, C γ ≥1;

[0021] Step 3_7: According to The style sparse coefficient matrix of the Gram matrix of the two-dimensional feature map re-aggregated from each three-dimensional feature map in The style sparse coefficient matrix of the Gram matrix of the two-dimensional feature map re-aggregated from each three-dimensional feature map in R is calculated. test The objective evaluation prediction value of style similarity is denoted as Q style ;

[0022] Step 3_8: According to Q content and Q style , calculate R test The overall visual objective evaluation prediction value is denoted as Q overall .

[0023] In the steps 1_3, CD z,o Solved by online dictionary learning method Get; where min() is the minimum function, the symbol “|| ||2” is the symbol for obtaining the L2 norm of the matrix, Y z,o express The matrix consisting of all column vectors in , Y z,o The dimension is 64×M, for The first column vector in , Also said N of the oth layer in the zth group c A column vector consisting of the pixel values ​​of all pixels in the first sub-block of all sub-blocks in the structured image. for The tth column vector in , for The Mth column vector in , Also said N of the oth layer in the zth group c The column vector composed of the pixel values ​​of all pixels in the Mth sub-block of all sub-blocks in the structured image, X z,o represents the content sparse coefficient matrix, X z,o The dimension is K×M, Represents X z,o The first column vector in , Represents X z,o The tth column vector in , Represents Xz,o The Mth column vector in , The dimensions of are all K×1, the symbol “[]” represents a vector or matrix, and “st” means “constrained by…”. It means that for any t, 1≤t≤M, the symbol “||||0” is the symbol for obtaining the L0 norm of the matrix, and τ is the error coefficient.

[0024] In the step 2_2, The process of obtaining is: Re-aggregate into dimensions The two-dimensional feature map of Then calculate The Gram matrix of Recalculate The row average is calculated The average value of all elements in each row of The style feature vector based on the Gram matrix in, The dimension is C γ ×C γ , the superscript “T” indicates the transpose of a vector or matrix.

[0025] In the steps 2_3, SD γ Solved by online dictionary learning method Get; where min() is the minimum function, the symbol "||||2" is the symbol for obtaining the L2 norm of the matrix, G γ express The matrix consisting of all column vectors in , G γ The dimension is C γ ×N s , for The first column vector in , Also said The style feature vector based on the Gram matrix of the first three-dimensional feature map belonging to the γth convolutional layer, for The jth column vector in , Also said The style feature vector based on the Gram matrix of the j-th 3D feature map belonging to the γ-th convolutional layer in , for The Nth s column vector, Also said The Nth convolutional layer in the γth convolutional layer sThe style feature vector based on the Gram matrix of the three-dimensional feature map, V γ represents the style sparse coefficient matrix, V γ The dimension is B γ ×N s , Indicates V γ The first column vector in , Indicates V γ The j-th column vector in , Indicates V γ The Nth s column vector, The dimensions are all B γ ×1, the symbol “[]” represents a vector or matrix, and “st” means “constrained by…”. It means that for any j, 1≤j≤N s , the symbol “||||0” is the symbol for obtaining the L0 norm of the matrix, and τ is the error coefficient.

[0026] In the step 3_3, Among them, (CD z,o ) + Indicates CD z,o The generalized inverse matrix of .

[0027] In the steps 3_4, Among them, the symbol “<>” represents the inner product operation symbol, the symbol “||||2” is the symbol for obtaining the L2 norm of the matrix, and η is a very small constant to prevent the denominator from being zero.

[0028] In the steps 3-6, Among them, (SD γ ) + Indicates SD γ The generalized inverse matrix of .

[0029] In the steps 3-7, Among them, the symbol “<>” represents the inner product operation symbol, the symbol “||||2” is the symbol for obtaining the L2 norm of the matrix, and η is a very small constant to prevent the denominator from being zero.

[0030] In the steps 3-8, Among them, ω1 is used to adjust Q content The weight of ω2 is used to adjust Q style The weight of .

[0031] Compared with the prior art, the advantages of the present invention are:

[0032] 1) The method of the present invention takes into account the influence of content structure fidelity on the quality of style transfer images, selects multiple original undistorted natural images as content images to form a content image training set, uses multi-scale Gaussian difference filtering to filter each content image in the content image training set to obtain structural images of different scales, and performs dictionary training operations on a set consisting of all sub-blocks in all structural images belonging to the same scale in an unsupervised learning manner to construct a multi-scale content dictionary. In the test phase, there is no need to calculate the content dictionary table, which avoids the complex machine learning training process, reduces the computational complexity, and does not need to predict the subjective evaluation value of each evaluation image, so that the method of the present invention is suitable for practical application occasions.

[0033] 2) The method of the present invention takes into account the impact of style similarity on the quality of style transferred images, selects multiple original artistic style images to form an artistic style image training set, and uses image deep structure and texture similarity measurement criteria to extract the three-dimensional feature maps of the five convolutional layers corresponding to each artistic style image in the artistic style image training set, thereby obtaining a style feature vector based on the Gram matrix of each three-dimensional feature map, and performs a dictionary training operation on the set of style feature vectors based on the Gram matrix of all three-dimensional feature maps belonging to the same convolutional layer in an unsupervised learning manner to construct a multi-scale style dictionary. In the test phase, there is no need to calculate the style dictionary table, which avoids the complex machine learning training process, reduces the computational complexity, and does not require the subjective evaluation value of each evaluation image to be known in advance, making the method of the present invention suitable for practical applications.

[0034] 3) In the testing phase, the method of the present invention obtains a content sparse coefficient matrix and a style sparse coefficient matrix according to the multi-scale content dictionary and the multi-scale style dictionary constructed in the training phase, and then calculates the objective quality evaluation prediction value of the style transfer image through the content sparse coefficient matrix and the style sparse coefficient matrix, so that the final calculated objective quality evaluation prediction value of the style transfer image has good consistency with the average subjective evaluation value. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 The figure is a general implementation block diagram of the method of the present invention. DETAILED DESCRIPTION

[0036] The present invention is further described in detail below with reference to the accompanying drawings.

[0037] The present invention proposes an objective evaluation method for style transfer image quality based on sparse representation, and its overall implementation block diagram is as follows: Figure 1 As shown, the method is characterized by comprising two processes: a training phase and a testing phase. The training phase comprises a multi-scale content dictionary training phase and a multi-scale style dictionary training phase.

[0038] The specific steps of the multi-scale content dictionary training phase are as follows:

[0039] Step 1_1: Select N c The original undistorted natural images constitute the content image training set, denoted as {IC i |1≤i≤N c}; where N c ≥1, in this embodiment, N c =10,IC i Indicates {IC i |1≤i≤N c The i-th content image in} represents the i-th original undistorted natural image. The symbol “{}” is a set representation symbol, i is a positive integer, 1≤i≤N c ,{IC i |1≤i≤N c} Each content image in has a width of WC and a height of HC.

[0040] Step 1_2: Use the existing multi-scale difference of Gaussian (DoG) technology to filter {IC i |1≤i≤N c} implements multi-scale structure separation for each content image in {IC i |1≤i≤N c}, and the structure images of different scales corresponding to each content image in {IC i |1≤i≤N c The set of structural images of different scales corresponding to all content images in} is recorded as Wherein, z and o are both positive integers, 1≤z≤Z, 1≤o≤O, Z represents the total number of groups in the Gaussian pyramid used in the multi-scale Gaussian difference filtering technology, O represents the total number of layers in each group in the Gaussian pyramid used in the multi-scale Gaussian difference filtering technology, and in this embodiment, Z=3, O=4, Indicates IC i The corresponding structural image of the oth layer in the zth group, The width is WC and the height is HC.

[0041] Step 1_3: Each structural image in non-overlapping sub-blocks of size 8×8; then N that belong to the same scale (i.e., the same layer in the same group) c All sub-blocks in the structured image form a set. N of the oth layer in the zth groupc The set of all sub-blocks in the structured image is recorded as Then the existing online dictionary learning (ODL) method is used to N of the same scale c The dictionary training operation is performed on the set of all sub-blocks in the structured image, and the N of the same scale c The content dictionary table corresponding to the set of all sub-blocks in the structured image will be The corresponding content dictionary obtained by performing the dictionary training operation is denoted as CD z,o Finally, we get The corresponding multi-scale content dictionary is denoted as {CD z,o |1≤z≤Z,1≤o≤O}; where WC and HC are both divisible by 8, 1≤t≤M, express N of the oth layer in the zth group c is a column vector consisting of the pixel values ​​of all pixels in the t-th sub-block of all sub-blocks in the structured image. The dimension is 64×1, CD z,o The dimension is 64×K, K represents the number of atoms in the set dictionary, K≥1, if the value of K is too large, over-clustering will occur, if the value of K is too small, under-clustering will occur, in this embodiment, K=256.

[0042] In this specific embodiment, in step 1_3, CD z,o Solved by existing online dictionary learning (ODL) method Get; where min() is the minimum function, the symbol "||||2" is the symbol for obtaining the L2 norm of the matrix, Y z,o express The matrix consisting of all column vectors in , Y z,o The dimension is 64×M, for The first column vector in , Also said N of the oth layer in the zth group c A column vector consisting of the pixel values ​​of all pixels in the first sub-block of all sub-blocks in the structured image. for The tth column vector in , for The Mth column vector in , Also said N of the oth layer in the zth group c The column vector composed of the pixel values ​​of all pixels in the Mth sub-block of all sub-blocks in the structured image, X z,o represents the content sparse coefficient matrix, X z,o The dimension is K×M, Represents X z,o The first column vector in , Represents X z,o The tth column vector in , Represents X z,o The Mth column vector in , The dimensions of are all K×1, the symbol “[]” represents a vector or matrix, and “st” means “constrained by…”. It means that for any t, 1≤t≤M, the symbol “||||0” is the symbol for obtaining the L0 norm of the matrix, τ is the error coefficient, and in this embodiment, τ is 0.1.

[0043] The specific steps of the multi-scale style dictionary training phase are as follows:

[0044] Step 2_1: Select N s The original artistic style images constitute the artistic style image training set, denoted as {IS j |1≤j≤N s}; where N s ≥1, in this embodiment, N s =2000,IS j Indicates {IS j |1≤j≤N s}, the symbol "{}" is a set representation symbol, j is a positive integer, 1≤j≤N s , I.S. j The width is WS j And the height is HS j ; Here, {IS j |1≤j≤N s The width and height of each artistic style image in} may be different, for example, the width and height of the first artistic style image are different from the width and height of the second artistic style image.

[0045] Step 2_2: Use the existing Deep Image Structure and Texture Similarity (DISTS) metric to measure {IS j |1≤j≤Ns} to generate each artistic style image in {IS j |1≤j≤N s The three-dimensional feature map of the five convolutional layers corresponding to each artistic style image in {IS j |1≤j≤N s The set of three-dimensional feature maps of the five convolutional layers corresponding to all artistic style images in} is recorded as Recalculate The style feature vector based on the Gram matrix of each three-dimensional feature map in The style feature vector based on the Gram matrix is ​​recorded as Among them, 1≤γ≤5, Indicates IS j The corresponding 3D feature map of the γth convolutional layer, The dimension is C γ represents the number of channels of the three-dimensional feature map generated by the γth convolutional layer in the image deep structure and texture similarity measurement criterion. In this embodiment, C 1 =64, C 2 =128, C 3 =256, C 4 =512, C 5 =512, Expressing support for IS j The height of the 3D feature map generated by the γth convolutional layer in the image deep structure and texture similarity metric is Height, Expressing support for IS j The width of the 3D feature map generated by the γth convolutional layer in the image deep structure and texture similarity metric, i.e. The width of The dimension is C γ ×1, and For calculation The intermediate variable.

[0046] In this specific embodiment, in step 2_2, The acquisition process is: using existing technology to Re-aggregate into dimensions The two-dimensional feature map of Then calculate The Gram matrix of Recalculate The row average is calculated The average value of all elements in each row of The style feature vector based on the Gram matrix in, The dimension is C γ ×C γ , the superscript “T” indicates the transpose of a vector or matrix.

[0047] Step 2_3: N belongs to the same convolutional layer s The style feature vectors based on the Gram matrix of the three-dimensional feature map form a set. N belongs to the γth convolutional layer s The set of style feature vectors based on the Gram matrix of a three-dimensional feature map is recorded as Then we use the existing online dictionary learning (ODL) method to N belongs to the same convolutional layer s The dictionary training operation is performed on the set of style feature vectors based on the Gram matrix of the three-dimensional feature map, and the N belongs to the same convolutional layer s The style dictionary table corresponding to the set of style feature vectors based on the Gram matrix of the three-dimensional feature map will be The corresponding style dictionary obtained by performing the dictionary training operation is denoted as SD γ Finally, we get The corresponding multi-scale style dictionary is denoted as {SD γ |1≤γ≤5}; where SD γ The dimension is C γ ×B γ , B γ represents the number of atoms in the dictionary set for the γth convolutional layer, B γ ≥1, B γ If the value of B is too large, clustering will occur. γ If the value of is too small, under-clustering will occur. 1 =256, B 2 =256, B 3 =512, B 4 =1024, B 5 =1024, i.e. SD 1 The dimension is 64×256, SD 2 The dimension is 128×256, SD 3 The dimension is 256×512, SD 4 The dimension is 512×1024, SD 5The dimensions are 512×1024.

[0048] In this specific embodiment, in step 2_3, SD γ Solved by online dictionary learning method Get; where min() is the minimum function, the symbol "||||2" is the symbol for obtaining the L2 norm of the matrix, G γ express The matrix consisting of all column vectors in , G γ The dimension is C γ ×N s , for The first column vector in , Also said The style feature vector based on the Gram matrix of the first three-dimensional feature map belonging to the γth convolutional layer, for The j-th column vector in , Also said The style feature vector based on the Gram matrix of the j-th 3D feature map belonging to the γ-th convolutional layer in , for The Nth s column vector, Also said The Nth convolutional layer in the γth convolutional layer s The style feature vector based on the Gram matrix of the three-dimensional feature map, V γ represents the style sparse coefficient matrix, V γ The dimension is B γ ×N s , Indicates V γ The first column vector in , Indicates V γ The j-th column vector in , Indicates V γ The Nth s column vector, The dimensions are all B γ ×1, the symbol “[]” represents a vector or matrix, and “st” means “constrained by…”. It means that for any j, 1≤j≤N s , the symbol “||||0” is the symbol for obtaining the L0 norm of the matrix, τ is the error coefficient, and in this embodiment, τ is taken as 0.1.

[0049] The specific steps of the testing phase are as follows:

[0050] Step 3_1: For any tested style transfer image, record it as R test ; and R test The corresponding original content image is denoted as IC org , R test The corresponding original artistic style image is denoted as IS org ; Among them, R test 、IC org The width is WC and the height is HC, IS org The width is WS org And the height is HS org .

[0051] Step 3_2: Follow the process of step 1_2 to obtain R in the same way test Corresponding structural images and ICs at different scales org Corresponding structural images of different scales, R test The corresponding set of structural images of different scales is recorded as IC org The corresponding set of structural images of different scales is recorded as in, Represents R test The corresponding structural image of the oth layer in the zth group, Indicates IC org The corresponding structural image of the oth layer in the zth group, The width is WC and the height is HC.

[0052] Step 3_3: Each structural image in non-overlapping sub-blocks of size 8×8; Each structural image in non-overlapping sub-blocks of size 8×8; then All sub-blocks in each structural image in form a set. The set of all sub-blocks in is recorded as and will All sub-blocks in each structural image in form a set. The set of all sub-blocks in is recorded as Then according to {CD z,o |1≤z≤Z,1≤o≤O}, get The content sparse coefficient matrix of each element in the set of all sub-blocks in the structure image of the corresponding scale in , that is, according to CD z,o , get The content of each element in the sparse coefficient matrix will be The sparse coefficient matrix of the content is recorded as Similarly, according to {CD z,o |1≤z≤Z,1≤o≤O}, get The content sparse coefficient matrix of each element in the set of all sub-blocks in the structure image of the corresponding scale in , that is, according to CD z,o , get The content of each element in the sparse coefficient matrix will be The sparse coefficient matrix of the content is recorded as in, express A column vector consisting of the pixel values ​​of all pixels in the t'th sub-block, The dimension is 64×1, express A column vector consisting of the pixel values ​​of all pixels in the t'th sub-block, The dimension is 64×1, The dimensions of are all K×1.

[0053] In this specific embodiment, in step 3_3, Among them, (CD z,o ) + Indicates CD z,o The generalized inverse matrix of .

[0054] Step 3_4: According to The content sparse coefficient matrix of each element in the set composed of all sub-blocks in each structural image in The content sparse coefficient matrix of each element in the set of all sub-blocks in each structural image in R is calculated. test The objective evaluation prediction value of content retention is denoted as Q content .

[0055] In this specific embodiment, in step 3_4, Among them, the symbol “<>” represents the inner product operation symbol, the symbol “||||2” is the symbol for obtaining the L2 norm of the matrix, and η is a very small constant to prevent the denominator from being zero. In this embodiment, η is taken as 0.02.

[0056] Step 3_5: Follow the process of step 2_2 to obtain R in the same way test The corresponding three-dimensional feature map and IS of the five convolutional layers org The corresponding three-dimensional feature map of the five convolutional layers is R test The corresponding set of three-dimensional feature maps of the five convolutional layers is recorded as ISorg The corresponding set of three-dimensional feature maps of the five convolutional layers is recorded as Then, using existing technology Each 3D feature map in is reaggregated into a dimension of C γ ×(HC γ ×WC γ )’s two-dimensional feature map, The re-aggregated two-dimensional feature map is recorded as Similarly, using existing technology Each 3D feature map in is reaggregated into a dimension of The two-dimensional feature map of The re-aggregated two-dimensional feature map is recorded as Recalculate Each three-dimensional feature map in is re-aggregated into the Gram matrix of the two-dimensional feature map, The Gram matrix of Similarly, calculation Each three-dimensional feature map in is re-aggregated into the Gram matrix of the two-dimensional feature map, The Gram matrix of Among them, 1≤γ≤5, Represents R test The corresponding 3D feature map of the γth convolutional layer, Indicates IS org The corresponding 3D feature map of the γth convolutional layer, The dimension is C γ ×HC γ ×WC γ , The dimension is C γ represents the number of channels of the three-dimensional feature map generated by the γth convolutional layer in the image deep structure and texture similarity measurement criterion. In this embodiment, C 1 =64, C 2 =128, C 3 =256, C 4 =512, C 5 =512,HC γ Indicates R test The height of the 3D feature map generated by the γth convolutional layer in the image deep structure and texture similarity metric, WC γ Indicates R test The width of the 3D feature map generated by the γth convolutional layer in the image deep structure and texture similarity metric, Expressing support for IS orgThe height of the 3D feature map generated by the γth convolutional layer in the image deep structure and texture similarity metric, Expressing support for IS org The width of the 3D feature map generated by the γth convolutional layer in the image deep structure and texture similarity metric, The dimensions are all C γ ×C γ , the superscript "T" indicates the transpose of a vector or matrix, and For calculation The intermediate variable, HC γ and WC γ For calculation The intermediate variable.

[0057] Step 3_6: According to {SD γ |1≤γ≤5}, get The style sparse coefficient matrix of the Gram matrix of the two-dimensional feature map re-aggregated from the three-dimensional feature map of the corresponding convolutional layer, that is, according to SD γ , get The style sparse coefficient matrix will be The style sparse coefficient matrix is ​​recorded as Similarly, according to {SD γ |1≤γ≤5}, get The style sparse coefficient matrix of the Gram matrix of the two-dimensional feature map re-aggregated from the three-dimensional feature map of the corresponding convolutional layer, that is, according to SD γ , get The style sparse coefficient matrix will be The style sparse coefficient matrix is ​​recorded as in, The dimensions are all B γ ×C γ , B γ ≥1, C γ ≥1; in this embodiment, B 1 =256, B 2 =256, B 3 =512, B 4 =1024, B 5 =1024, C 1 =64, C 2 =128, C 3 =256, C 4 =512, C 5 =512.

[0058] In this specific embodiment, in steps 3-6, Among them, (SDγ ) + Indicates SD γ The generalized inverse matrix of .

[0059] Step 3_7: According to The style sparse coefficient matrix of the Gram matrix of the two-dimensional feature map re-aggregated from each three-dimensional feature map in The style sparse coefficient matrix of the Gram matrix of the two-dimensional feature map re-aggregated from each three-dimensional feature map in R is calculated. test The objective evaluation prediction value of style similarity is denoted as Q style .

[0060] In this specific embodiment, in steps 3-7, Among them, the symbol “<>” represents the inner product operation symbol, the symbol “||||2” is the symbol for obtaining the L2 norm of the matrix, and η is a very small constant to prevent the denominator from being zero. In this embodiment, η is taken as 0.02.

[0061] Step 3_8: According to Q content and Q style , calculate R test The overall visual objective evaluation prediction value is denoted as Q overall .

[0062] In this specific embodiment, in steps 3-8, Among them, ω1 is used to adjust Q content The weight of ω2 is used to adjust Q style In this embodiment, ω1 takes the value of 1 and ω2 takes the value of 1.

[0063] In order to further illustrate the feasibility and effectiveness of the method of the present invention, the method of the present invention was tested.

[0064] In the experiment, the method of the present invention was used to test the style transfer image quality evaluation database established by the Multimedia Laboratory of Ningbo University. The style transfer image quality evaluation database contains 150 pairs of content images and artistic style images. Through 8 different style transfer algorithms, 1200 style transfer images of different qualities were obtained. Using the existing subjective evaluation method, three average subjective score differences of each style transfer image in the style transfer image quality evaluation database were obtained, including the average subjective score difference of content preservation, the average subjective score difference of style similarity, and the average subjective score difference of overall vision.

[0065] In the experiment, two commonly used objective parameters for evaluating image quality evaluation methods are used as evaluation indicators, namely, Spearman rank order correlation coefficient (SROCC) and Kendall rank-order correlation coefficient (KROCC). These two evaluation indicators reflect the monotonicity of the prediction.

[0066] Table 1 shows the Kendall correlation coefficient and Spearman correlation coefficient between the objective evaluation prediction value of the content preservation of the style migration image obtained by the method of the present invention and the difference between the average subjective score of the content preservation.

[0067] Table 1 Kendall correlation coefficient and Spearman correlation coefficient between the objective evaluation prediction value of content preservation of style migration images obtained by the method of the present invention and the difference between the average subjective score of content preservation

[0068] Kendall correlation coefficient Spearman correlation coefficient Method of the present invention 0.6731 0.7867

[0069] Table 2 shows the Kendall correlation coefficient and Spearman correlation coefficient between the objective evaluation prediction value of style similarity of the style transfer image obtained by the method of the present invention and the difference between the average subjective score of style similarity.

[0070] Table 2 Kendall correlation coefficient and Spearman correlation coefficient between the objective evaluation prediction value of style similarity of style transfer images obtained by the method of the present invention and the difference between the average subjective score of style similarity

[0071] Kendall correlation coefficient Spearman correlation coefficient Method of the present invention 0.4847 0.6038

[0072] Table 3 shows the Kendall correlation coefficient and Spearman correlation coefficient between the overall visual objective evaluation prediction value and the overall visual average subjective score difference of the style transfer image obtained by the method of the present invention.

[0073] Table 3 Kendall correlation coefficient and Spearman correlation coefficient between the overall visual objective evaluation prediction value and the overall visual average subjective score difference of the style transfer image obtained by the method of the present invention

[0074] Kendall correlation coefficient Spearman correlation coefficient Method of the present invention 0.4841 0.6010

[0075] It can be seen from Tables 1, 2 and 3 that the correlation between the objective evaluation prediction value of the content preservation of the style transfer image obtained by the method of the present invention and the difference between the average subjective score of the content preservation, the correlation between the objective evaluation prediction value of the style similarity of the style transfer image and the difference between the average subjective score of the style similarity, and the correlation between the overall visual objective evaluation prediction value of the style transfer image and the difference between the overall visual average subjective score are all relatively high, indicating that the objective evaluation results are relatively consistent with the results of subjective perception of the human eye, which is sufficient to illustrate the effectiveness of the method of the present invention.

Claims

1. An objective evaluation method for style transfer image quality based on sparse representation, characterized by It includes two processes: training phase and testing phase. The training phase includes multi-scale content dictionary training phase and multi-scale style dictionary training phase. The specific steps of the multi-scale content dictionary training phase are as follows: Step 1_1: Select N c The original undistorted natural images constitute the content image training set, denoted as {IC i |1≤i≤N c }; where N c ≥1, IC i Indicates {IC i |1≤i≤N c }, i is a positive integer, 1≤i≤N c ,{IC i |1≤i≤N c } Each content image in has a width of WC and a height of HC; Step 1_2: Use multi-scale Gaussian difference filtering technology to {IC i |1≤i≤N c } implements multi-scale structure separation for each content image in {IC i |1≤i≤N c }, and the structure images of different scales corresponding to each content image in {IC i |1≤i≤N c The set of structural images of different scales corresponding to all content images in} is recorded as Wherein, z and o are both positive integers, 1≤z≤Z, 1≤o≤O, Z represents the total number of groups in the Gaussian pyramid used in the multi-scale Gaussian difference filtering technology, O represents the total number of layers in each group in the Gaussian pyramid used in the multi-scale Gaussian difference filtering technology, Indicates IC i The corresponding structural image of the oth layer in the zth group, The width is WC and the height is HC; Step 1_3: Each structural image in non-overlapping sub-blocks of size 8×8; then N of the same scale c All sub-blocks in the structured image form a set. N of the oth layer in the zth group c The set of all sub-blocks in the structured image is recorded as Then use the online dictionary learning method to N of the same scale c The dictionary training operation is performed on the set of all sub-blocks in the structured image, and the N of the same scale c The content dictionary table corresponding to the set of all sub-blocks in the structured image will be The corresponding content dictionary obtained by performing the dictionary training operation is denoted as CD z,o Finally, we get The corresponding multi-scale content dictionary is denoted as {CD z,o |1≤z≤Z,1≤o≤O}; where WC and HC are both divisible by 8, 1≤t≤M, express N of the oth layer in the zth group c is a column vector consisting of the pixel values ​​of all pixels in the t-th sub-block of all sub-blocks in the structured image. The dimension is 64×1, CD z,o The dimension is 64×K, where K represents the number of atoms in the set dictionary, K≥1; The specific steps of the multi-scale style dictionary training phase are as follows: Step 2_1: Select N s The original artistic style images constitute the artistic style image training set, denoted as {IS j |1≤j≤N s }; where N s ≥1, IS j Indicates {IS j |1≤j≤N s }, j is a positive integer, 1≤j≤N s , I.S. j The width is WS j And the height is HS j ; Step 2_2: Use the image deep structure and texture similarity measurement criteria to measure {IS j |1≤j≤N s } to generate each artistic style image in {IS j |1≤j≤N s The three-dimensional feature map of the five convolutional layers corresponding to each artistic style image in {IS j |1≤j≤N s The set of three-dimensional feature maps of the five convolutional layers corresponding to all artistic style images in} is recorded as Recalculate The style feature vector based on the Gram matrix of each three-dimensional feature map in The style feature vector based on the Gram matrix is ​​recorded as Among them, 1≤γ≤5, Indicates IS j The corresponding 3D feature map of the γth convolutional layer, The dimension is C γ represents the number of channels of the 3D feature map generated by the γth convolutional layer in the image deep structure and texture similarity metric, Expressing support for IS j The height of the 3D feature map generated by the γth convolutional layer in the image deep structure and texture similarity metric is Height, Expressing support for IS j The width of the 3D feature map generated by the γth convolutional layer in the image deep structure and texture similarity metric, i.e. The width of The dimension is C γ ×1; Step 2_3: N belongs to the same convolutional layer s The style feature vectors based on the Gram matrix of the three-dimensional feature map form a set. N belongs to the γth convolutional layer s The set of style feature vectors based on the Gram matrix of a three-dimensional feature map is recorded as Then use the online dictionary learning method to N belongs to the same convolutional layer s The dictionary training operation is performed on the set of style feature vectors based on the Gram matrix of the three-dimensional feature map, and the N belongs to the same convolutional layer s The style dictionary table corresponding to the set of style feature vectors based on the Gram matrix of the three-dimensional feature map will be The corresponding style dictionary obtained by performing the dictionary training operation is denoted as SD γ Finally, we get The corresponding multi-scale style dictionary is denoted as {SD γ |1≤γ≤5}; where SD γ The dimension is C γ ×B γ , B γ represents the number of atoms in the dictionary set for the γth convolutional layer, B γ ≥1; The specific steps of the testing phase are as follows: Step 3_1: For any tested style transfer image, record it as R test ; and R test The corresponding original content image is denoted as IC org , R test The corresponding original artistic style image is denoted as IS org ; Among them, R test 、IC org The width is WC and the height is HC, IS org The width is WS org And the height is HS org ; Step 3_2: Follow the process of step 1_2 to obtain R in the same way test Corresponding structural images and ICs at different scales org Corresponding structural images of different scales, R test The corresponding set of structural images of different scales is recorded as IC org The corresponding set of structural images of different scales is recorded as in, Represents R test The corresponding structural image of the oth layer in the zth group, Indicates IC org The corresponding structural image of the oth layer in the zth group, The width is WC and the height is HC; Step 3_3: Each structural image in non-overlapping sub-blocks of size 8×8; Each structural image in non-overlapping sub-blocks of size 8×8; then All sub-blocks in each structural image in form a set. The set of all sub-blocks in is recorded as and will All sub-blocks in each structural image in form a set. The set of all sub-blocks in is recorded as Then according to {CD z,o |1≤z≤Z,1≤o≤O}, get The content sparse coefficient matrix of each element in the set composed of all sub-blocks in the structural image of the corresponding scale in is The sparse coefficient matrix of the content is recorded as Similarly, according to {CD z,o |1≤z≤Z,1≤o≤O}, get The content sparse coefficient matrix of each element in the set composed of all sub-blocks in the structural image of the corresponding scale in is The sparse coefficient matrix of the content is recorded as in, express A column vector consisting of the pixel values ​​of all pixels in the t'th sub-block, The dimension is 64×1, express A column vector consisting of the pixel values ​​of all pixels in the t'th sub-block, The dimension is 64×1, The dimensions of are all K×1; Step 3_4: According to The content sparse coefficient matrix of each element in the set composed of all sub-blocks in each structural image in The content sparse coefficient matrix of each element in the set of all sub-blocks in each structural image in R is calculated. test The objective evaluation prediction value of content retention is denoted as Q content ; Step 3_5: Follow the process of step 2_2 to obtain R in the same way test The corresponding three-dimensional feature map and IS of the five convolutional layers org The corresponding three-dimensional feature map of the five convolutional layers is R test The corresponding set of three-dimensional feature maps of the five convolutional layers is recorded as IS org The corresponding set of three-dimensional feature maps of the five convolutional layers is recorded as Then Each 3D feature map in is reaggregated into a dimension of C γ ×(HC γ ×WC γ )’s two-dimensional feature map, The re-aggregated two-dimensional feature map is recorded as Likewise, Each 3D feature map in is reaggregated into a dimension of The two-dimensional feature map of The re-aggregated two-dimensional feature map is recorded as Recalculate Each three-dimensional feature map in is re-aggregated into the Gram matrix of the two-dimensional feature map, The Gram matrix of Similarly, calculation Each three-dimensional feature map in is re-aggregated into the Gram matrix of the two-dimensional feature map, The Gram matrix of Among them, 1≤γ≤5, Represents R test The corresponding 3D feature map of the γth convolutional layer, Indicates IS org The corresponding 3D feature map of the γth convolutional layer, The dimension is C γ ×HC γ ×WC γ , The dimension is C γ represents the number of channels of the 3D feature map generated by the γth convolutional layer in the image deep structure and texture similarity metric, HC γ Indicates R test The height of the 3D feature map generated by the γth convolutional layer in the image deep structure and texture similarity metric, WC γ Indicates R test The width of the 3D feature map generated by the γth convolutional layer in the image deep structure and texture similarity metric, Expressing support for IS org The height of the 3D feature map generated by the γth convolutional layer in the image deep structure and texture similarity metric, Expressing support for IS org The width of the 3D feature map generated by the γth convolutional layer in the image deep structure and texture similarity metric, The dimensions are all C γ ×C γ , the superscript "T" indicates the transpose of a vector or matrix; Step 3_6: According to {SD γ |1≤γ≤5}, get The style sparse coefficient matrix of the Gram matrix of the two-dimensional feature map of the three-dimensional feature map of the corresponding convolution layer in is re-aggregated, and the The style sparse coefficient matrix is ​​recorded as Similarly, according to {SD γ |1≤γ≤5}, get The style sparse coefficient matrix of the Gram matrix of the two-dimensional feature map of the three-dimensional feature map of the corresponding convolution layer in is re-aggregated, and the The style sparse coefficient matrix is ​​recorded as in, The dimensions are all B γ ×C γ , B γ ≥1, C γ ≥1; Step 3_7: According to The style sparse coefficient matrix of the Gram matrix of the two-dimensional feature map re-aggregated from each three-dimensional feature map in The style sparse coefficient matrix of the Gram matrix of the two-dimensional feature map re-aggregated from each three-dimensional feature map in R is calculated. test The objective evaluation prediction value of style similarity is denoted as Q style ; Step 3_8: According to Q content and Q style , calculate R test The overall visual objective evaluation prediction value is denoted as Q overall .

2. According to claim 1, a method for objectively evaluating the quality of style transfer images based on sparse representation is characterized in that In the steps 1_3, CD z,o Solved by online dictionary learning method Get; where min() is the minimum function, the symbol "|| ||2" is the symbol for obtaining the L2 norm of the matrix, Y z,o express The matrix consisting of all column vectors in , Y z,o The dimension is 64×M, for The first column vector in , Also said N of the oth layer in the zth group c A column vector consisting of the pixel values ​​of all pixels in the first sub-block of all sub-blocks in the structured image. for The tth column vector in , for The Mth column vector in , Also said N of the oth layer in the zth group c The column vector composed of the pixel values ​​of all pixels in the Mth sub-block of all sub-blocks in the structured image, X z,o represents the content sparse coefficient matrix, X z,o The dimension is K×M, Represents X z,o The first column vector in , Represents X z,o The tth column vector in , Represents X z,o The Mth column vector in , The dimensions of are all K×1, the symbol "[]" represents a vector or matrix, and "st" means "constrained by...". It means that for any t, 1≤t≤M, the symbol "|| ||0" is the symbol for obtaining the L0 norm of the matrix. is the error coefficient.

3. The objective evaluation method of style transfer image quality based on sparse representation according to claim 1 or 2, characterized in that In the step 2_2, The process of obtaining is: Re-aggregate into dimensions The two-dimensional feature map of Then calculate The Gram matrix of Recalculate The row average is calculated The average value of all elements in each row of The style feature vector based on the Gram matrix in, The dimension is C γ ×C γ , the superscript "T" indicates the transpose of a vector or matrix.

4. The objective evaluation method of style transfer image quality based on sparse representation according to claim 3 is characterized in that In the steps 2_3, SD γ Solved by online dictionary learning method Get; where min() is the minimum function, the symbol "|| ||2" is the symbol for obtaining the L2 norm of the matrix, G γ express The matrix consisting of all column vectors in , G γ The dimension is C γ ×N s , for The first column vector in , Also said The style feature vector based on the Gram matrix of the first three-dimensional feature map belonging to the γth convolutional layer, for The jth column vector in , Also said The style feature vector based on the Gram matrix of the j-th 3D feature map belonging to the γ-th convolutional layer in , for The Nth s column vector, Also said The Nth convolutional layer in the γth convolutional layer s The style feature vector based on the Gram matrix of the three-dimensional feature map, V γ represents the style sparse coefficient matrix, V γ The dimension is B γ ×N s , Indicates V γ The first column vector in , Indicates V γ The jth column vector in , Indicates V γ The Nth s column vector, The dimensions are all B γ ×1, the symbol "[]" represents a vector or matrix, and "st" means "constrained by...", It means that for any j, 1≤j≤N s , the symbol "|| ||0" is the symbol for obtaining the L0 norm of the matrix, and τ is the error coefficient.

5. The objective evaluation method of style transfer image quality based on sparse representation according to claim 4 is characterized in that In the step 3_3, Among them, (CD z,o ) + Indicates CD z,o The generalized inverse matrix of .

6. The objective evaluation method of style transfer image quality based on sparse representation according to claim 1, characterized in that In the steps 3_4, Among them, the symbol "<>" represents the inner product operation symbol, the symbol "|| ||2" is the symbol for obtaining the L2 norm of the matrix, and η is a very small constant to prevent the denominator from being zero.

7. The objective evaluation method of style transfer image quality based on sparse representation according to claim 1, characterized in that In the steps 3-6, Among them, (SD γ ) + Indicates SD γ The generalized inverse matrix of .

8. The objective evaluation method of style transfer image quality based on sparse representation according to claim 1, characterized in that In the steps 3-7, Among them, the symbol "<>" represents the inner product operation symbol, the symbol "|| ||2" is the symbol for obtaining the L2 norm of the matrix, and η is a very small constant to prevent the denominator from being zero.

9. The objective evaluation method of style transfer image quality based on sparse representation according to claim 1, characterized in that In the steps 3-8, Among them, ω1 is used to adjust Q content The weight of ω2 is used to adjust Q style The weight of .

Images