A deep and shallow layer full-reference image quality evaluation method based on visual perception

Abstract

The present application relates to the technical field of image processing, and particularly relates to a deep and shallow layer full-reference image quality evaluation method based on visual perception, comprising the following steps: acquiring reference images and distorted images with uniform size; extracting five layers of feature maps respectively, the first three layers being shallow layer feature maps, and the last two layers being deep layer feature maps; performing feature enhancement on the shallow layer feature maps and using Wasserstein distance to calculate three shallow layer feature quality scores; performing self-attention enhancement on the deep layer feature maps and calculating two layers of perception differences between the reference images and the distorted images, sequentially processing through an attention aggregation module, a splicing module and a fusion module, and using weighted scores and pixel prediction to obtain deep layer feature quality scores; S6, integrating the three shallow layer feature quality scores and one deep layer feature quality score through learnable parameters to obtain deep and shallow layer full-reference image quality scores. The method combines human visual perception and deep learning to realize efficient evaluation of image quality.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a deep and shallow layer full reference image quality evaluation method based on visual perception. BACKGROUND

[0002] Image quality assessment is a core issue in the field of image processing and computer vision, which plays an important role in image / video coding, super-resolution reconstruction and image / video visual quality improvement.

[0003] At present, the disclosed full reference image quality assessment method estimates the quality score of the distorted image by comparing the visual differences between two images. Such algorithms can not only be used to evaluate the effect of image compression coding, but also be used as a test benchmark for image processing algorithms or as an optimization target for image processing tasks. In the prior art, a disclosed full reference image quality objective evaluation method compares the visual differences between two images, performs deep fusion processing based on visual multiple features, and does not comprehensively consider the low-level visual features and high-level visual features of the image and focus on the important regions in the image.

[0004] Therefore, as for the important regions of the image, how to effectively extract low-level visual features and high-level visual features and design a targeted processing method for visual features of different layers to construct an evaluation index more consistent with the visual characteristics of the human eye is a technical problem that needs to be solved by those skilled in the art. SUMMARY

[0005] Therefore, the present application designs a deep and shallow layer full reference image quality evaluation method based on visual perception, aiming to solve the problem of how to effectively extract low-level visual features and high-level visual features and design a targeted processing method for visual features of different layers, and finally construct an evaluation index method more consistent with the visual characteristics of the human eye.

[0006] In order to achieve the above purpose, the present application adopts the following technical scheme:

[0007] A deep and shallow layer full reference image quality evaluation method based on visual perception is disclosed, comprising the following steps:

[0008] S1, obtaining a pair of reference images and distorted images with uniform size;

[0009] S2, extracting five layers of feature maps based on the reference image and the distorted image, respectively, wherein the first three layers of feature maps are defined as shallow feature maps, and the last two layers of feature maps are defined as deep feature maps;

[0010] S3, respectively performing feature enhancement on the first three layers of shallow feature maps;

[0011] S4, the Wasserstein distance is used to measure the difference between the reference image and the distorted image and the same layer shallow feature, and three shallow feature quality scores are calculated;

[0012] S5, the self-attention enhancement is performed on the deep feature maps of the last two layers, and the image perception difference between the reference image and the distorted image is calculated to measure the degradation degree of the same layer deep feature map; the image perception difference between the two layers of deep features is processed through an attention aggregation module, a splicing module and a fusion module in sequence, and the weighted score and the pixel prediction are used to obtain the fused deep feature quality score;

[0013] S6, the three shallow feature quality scores and one deep feature quality score are integrated through the four learnable parameters in the deep learning model to obtain the final deep and shallow full reference image quality score.

[0014] Further, in S1, the image size of the obtained reference image and distorted image pair is unified to 224x224 pixels, and the image size unification method is to use image processing software or library to unify the image size.

[0015] Further, in S2, it includes:

[0016] S21, the size-unified reference image and distorted image are respectively input into the visual geometry group network model VGG-16;

[0017] S22, the visual geometry group network model VGG-16 extracts the first three layers of shallow feature maps of the reference image and the distorted image based on the texture, color and shape of the image;

[0018] S23, the visual geometry group network model VGG-16 extracts the last two layers of deep feature maps of the reference image and the distorted image based on the high-level semantic information of the image.

[0019] Further, in S3, the first three layers of shallow feature maps of the reference image and the distorted image are respectively enhanced by using SENet, and the process is as follows:

[0020] S31, the SENet performs compression operation on the input shallow feature map along the spatial dimension to obtain the corresponding compression vector, and the expression is as follows:

[0021]

[0022] Wherein, c represents a specific feature channel, u c is the input shallow feature map of the feature channel c, F sq represents the compression operation; z cThe compressed vector of the input shallow feature map of the feature channel c has a dimension of 1*1*C, C is a real number with a global receptive field, which ensures that the dimension of the output matches the number of input feature channels; H and W are the length and width of the input shallow feature map, respectively;

[0023] In S32, the assigned weight value of the compressed shallow feature map output channel is identified through the Excitation operation, and the expression is:

[0024] s=F ex (z, W)=σ(g(z, W))=σ(W2δ(W1z))

[0025] wherein s is the assigned weight value of the compressed shallow feature map output channel, F ex represents the Excitation operation, z is the compressed vector of the shallow feature map, W is the general term of the matrix of the assigned weight value of the compressed shallow feature map output channel, wherein W1 and W2 are the weight matrices assigned by the first two full connections, and the weight matrices are all three-dimensional matrices; σ represents a sigmoid function, before the sigmoid function, W1 is multiplied by z to realize the first full connection; after the Relu layer processing, W2 is multiplied to complete the second full connection;

[0026] The two full connections are used to reduce the number of channels and the amount of calculation, the dimension of W1 is 1*1*C / r, r is a scaling parameter, and the dimension of W1 multiplied by z is 1*1*C / r; after multiplication, the dimension is unchanged after the Relu layer processing; then W2 is multiplied to complete the second full connection, the dimension of W2 is 1*1*C*C / r, and the output dimension after the second full connection is 1*1*C;

[0027] In S33, the neural network multiplies the input shallow feature map and the assigned weight value of the feature channel to obtain a feature-enhanced shallow feature map.

[0028] The expression of the feature-enhanced shallow feature map is:

[0029]

[0030] wherein, is the feature-enhanced shallow feature map, F scale is the feature enhancement operation, u c , s c are the input shallow feature map of the feature channel c and the assigned weight value of the feature channel c, respectively.

[0031] Further, in S4, the calculation process of each shallow feature quality score is:

[0032] S41, convert the shallow feature maps of the same layer of the reference image and the distorted image into one-dimensional feature vectors and perform normalization processing to obtain normalized probability distribution feature maps;

[0033] S42, calculate the WSD value between the reference image and the distorted image based on the two normalized probability distribution feature maps using the Wasserstein distance;

[0034] S43, map the calculated WSD value to the image quality evaluation score to evaluate the degradation degree of the distorted image.

[0035] Further, in S41, the normalized probability distribution feature map is expressed as:

[0036]

[0037] where P is the normalized probability distribution feature map, f is the shallow feature map after feature enhancement, C is the feature enhancement channel, H and W are the length and width of the shallow feature map after feature enhancement, and f i represents a single feature element in the shallow feature map after feature enhancement, and i takes a value ranging from 1 to CxHxW;

[0038] In S42, the Wasserstein distance is used to calculate the WSD value between the shallow feature image pairs of the same layer of the reference image and the distorted image, and the expression is:

[0039]

[0040] where X and Y represent the normalized probability distribution feature maps of the reference image and the distorted image, respectively, and represent the cumulative probability distribution functions of X and Y, respectively, which are calculated based on the normalized probability distribution P of the reference image and the distorted image; WSD is the image degradation distance between the two normalized probability distribution feature maps X and Y.

[0041] Further, S5 includes:

[0042] S51, perform self-attention enhancement on the deep feature maps of the last two layers of the reference image and the distorted image to obtain enhanced deep feature maps;

[0043] S52, calculate the perceptual difference of the image degradation degree between the reference image and the distorted image based on the enhanced deep feature maps of the reference image and the distorted image;

[0044] S53, input the perceptual difference of the two layers into the attention aggregation module for batch normalization processing to obtain the aggregated deep feature map;

[0045] S54, output the aggregated deep feature map to a splicing module for preliminary processing, discard layer processing and connection input to obtain a finally spliced deep feature map;

[0046] S55, input the finally spliced deep feature map to a fusion module for fusion operation to obtain a fused deep feature map;

[0047] S56, based on the fused deep feature map, use a weighted score and pixel prediction to obtain a final deep feature quality score.

[0048] Further, in S51, the self-attention enhancement includes normalization processing for capturing internal correlation information of the original deep feature map and performing feature enhancement on the feature map;

[0049] The enhanced deep feature map expression is:

[0050]

[0051] Wherein, Q, K, V represent query vector, key vector and value matrix vector in the self-attention mechanism respectively, and Softmax is a normalization function; dk is the dimension of the key vector, used for scaling dot product to avoid the problem of gradient disappearance when the dimension is large.

[0052] Further, in S51, the deep feature map expression of the i-th layer after self-attention enhancement is:

[0053] f i =Attn(F i W q ,F i W k ,F i W v )+F i

[0054] Wherein, f i is the deep feature map of the i-th layer after self-attention enhancement, F i is the i-th layer deep feature map before self-attention enhancement, and i is 4 or 5; W q , W k and W v represent the weights of the query vector, the key vector and the value matrix vector in the self-attention enhancement;

[0055] In S52, the perceptual difference calculation formula between the i-th layer deep feature map of the reference image and the distorted image after self-attention enhancement is:

[0056]

[0057] Wherein, a perceived difference between the i-th layer deep feature map of the reference image and the i-th layer deep feature map of the distorted image after feature enhancement, respectively, are the deep feature maps of the reference image and the distorted image after self-attention enhancement;

[0058] In S53, the expression of the aggregated deep feature map is:

[0059]

[0060] In S54, the expression of the preliminarily processed deep feature map is:

[0061]

[0062] where BatchNorm represents batch normalization processing, is the aggregated deep feature map, is the preliminarily processed deep feature map;

[0063] The expression of the i-th layer deep feature map after dropout layer processing is:

[0064]

[0065] where Dropout represents the dropout layer operation, is the preliminarily processed i-th layer deep feature map; is the aggregated i-th layer deep feature map, is the i-th layer deep feature map after dropout layer processing;

[0066] The expression of the final spliced deep feature map after input connection is:

[0067]

[0068] where, is the final spliced deep feature map after input connection, respectively, are the 4th layer and 5th layer deep feature maps after dropout layer processing;

[0069] In S55, the expression of the fused deep feature map is:

[0070]

[0071] where F fusion is the fused deep feature map, and the fusion process includes 3X3 convolution, Relu activation function, two consecutive 3X3 convolution, and Relu function output on the final spliced deep feature map

[0072] In S56, the expression of the final deep feature quality score is: ​

[0073] d 45 = Conv(F fusion ) o Sigmoid(Conv(F fusion ))

[0074] wherein d 45 is the final deep feature quality score, F fusion is the fused deep feature map, which is input into the dual-branch prediction module, one branch is used for pixel prediction, and the other branch is used for weighted score prediction; o represents dot product.

[0075] Further, in S6, the final deep and shallow layer full-reference image quality score expression of the integrated deep learning model is:

[0076] D = a1d1 + a2d2 + a3d3 + a 45 d 45

[0077] wherein a1, a2, a3, a 45 are respectively four learnable parameters in the deep learning model, indicating the key degree of different layers to the final quality evaluation score, d1, d2, d3 are respectively the quality evaluation scores of the first three shallow layer feature maps, and d 45 is the final deep layer feature quality score.

[0078] According to the above technical solution, compared with the prior art, the present application has the following beneficial effects:

[0079] (1) The feature map is extracted by VGG-16 and logically divided to obtain shallow and deep feature maps, which provides a basis for subsequent targeted feature processing, so that we can start from the deep and shallow layer feature map to perform quality evaluation related operations;

[0080] (2) For shallow features, SENet is used for enhancement processing, its self-learning characteristics can accurately identify the importance of each convolution feature channel and assign reasonable weights to different convolution channels, so that the neural network focuses on the feature channel related to the task, avoiding the influence of irrelevant channels, which helps to highlight the feature information important to quality evaluation, and lays a good foundation for subsequent accurate calculation of shallow feature quality score;

[0081] (3) The degradation distance of the same layer shallow feature map is perceived using the Wasserstein distance, which prepares for the subsequent quality score, and this process reflects the image quality situation from the HVS consideration of the shallow features; for the deep features, the self-attention mechanism is first used for enhancement, so that it can better capture the internal correlation information, and then the perceptual difference between the two deep feature maps after feature enhancement is calculated to clarify the degradation degree of the distorted image, and a series of operations such as attention aggregation module, splicing module, fusion module and the like are sequentially passed through to fully integrate the features at different levels, and finally the weighted score and pixel prediction are used to obtain the quality score of the deep features, so that this all-round processing method can more deeply, comprehensively and accurately evaluate the quality level of the deep features, and mine the key information about the image quality contained in the deep features.

[0082] (4) The shallow feature quality score and the deep feature quality score are integrated through the learnable parameters, and under the specific calculation method and parameter adjustment, the mutual fusion and synergistic effect of the quality scores of the two different levels of features are realized, so that the final score which can comprehensively reflect the overall situation is obtained, so as to comprehensively evaluate the performance level of the related features at the overall level, so that the final quality evaluation result is more comprehensive and accurate, and closer to the actual quality situation of the image.

[0083] (5) The deep and shallow layer full reference image quality evaluation method based on visual perception of the present application can realize the quality score evaluation and prediction of the distorted image closer to the human eye perception through the steps of layer-by-layer feature extraction, feature enhancement of the deep and shallow features, and finally full reference image quality score calculation and integration, and provides an effective technical means for accurately evaluating the image quality. BRIEF DESCRIPTION OF DRAWINGS

[0084] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description are only embodiments of the present application, and those skilled in the art can obtain other drawings according to the provided drawings without any creative labor.

[0085] Figure 1 A flow chart of a deep and shallow layer full reference image quality evaluation method based on visual perception provided by the present application;

[0086] Figure 2 An architecture diagram of a deep and shallow layer full reference image quality evaluation method based on visual perception provided by the present application; DETAILED DESCRIPTION

[0087] The following will be combined with the drawings of the embodiments of the present application to Figures 1-2The technical solutions in the embodiments of the present application are clearly and completely described. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the protection scope of the present application.

[0088] The embodiment of the present application provides a deep-shallow full-reference image quality evaluation method based on visual perception, as shown in the formula (1), comprising the following steps: Figure 1

[0089] S1, acquiring a pair of reference images and distorted images with unified sizes;

[0090] S2, extracting five layers of feature maps based on the reference images and the distorted images respectively, wherein the first three layers of feature maps are defined as shallow feature maps, and the last two layers of feature maps are defined as deep feature maps;

[0091] S3, respectively performing feature enhancement on the first three layers of shallow feature maps;

[0092] S4, using the Wasserstein distance to measure the difference between the reference images and the distorted images with the same layer of shallow feature maps based on the first three layers of shallow feature maps after feature enhancement, and calculating three shallow feature quality scores;

[0093] S5, performing self-attention enhancement on the last two layers of deep feature maps, and calculating the image perception difference between the reference images and the distorted images with the same layer of deep feature maps, for measuring the degradation degree of the same layer of deep feature maps; the image perception difference between the two layers of deep feature maps is processed through an attention aggregation module, a splicing module and a fusion module in sequence, and a weighted score and a pixel prediction are used to obtain a fused deep feature quality score;

[0094] S6, integrating the three shallow feature quality scores and one deep feature quality score through four learnable parameters in a deep learning model, to obtain a final deep-shallow full-reference image quality score.

[0095] In the embodiment, in S1, the image sizes of the reference images and the distorted images are unified to 224x224 pixel size, and the image size unification manner is to use image processing software or library to unify the image size.

[0096] In the embodiment, in S2, it includes:

[0097] S21, inputting the reference images and the distorted images with unified sizes into a visual geometry group network model VGG-16 respectively;

[0098] ​S22, the visual geometry group network model VGG-16 extracts the first three layers of shallow feature maps of the reference image and the distorted image based on the texture, color and shape of the image;

[0099] S23, the visual geometry group network model VGG-16 extracts the last two layers of deep feature maps of the reference image and the distorted image based on the high-level semantic information of the image.

[0100] In this embodiment, in S3, the first three layers of shallow feature maps of the reference image and the distorted image are respectively enhanced by SENet, and the process is as follows:

[0101] S31, SENet performs compression operation on the input shallow feature map along the spatial dimension to obtain the corresponding compressed vector, and the expression is as follows:

[0102]

[0103] Wherein, c represents a specific feature channel, u c is the input shallow feature map of the feature channel c, F sq represents the compression operation; z c is the vector of the compressed shallow feature map input by the feature channel c, which has a dimension of 1*1*C, C is a real number with a global receptive field, which ensures that the dimension of the output matches the number of feature channels of the input; H and W are the length and width of the input shallow feature map respectively;

[0104] S32, the allocated weight of the output channel of the compressed shallow feature map is identified by Excitation operation, and the expression is as follows:

[0105] s=F ex (z,W)=σ(g(zW))=σ(W2δ(W1z))

[0106] Wherein, s is the allocated weight of the output channel of the compressed shallow feature map, F ex represents the Excitation operation, z is the compressed vector of the shallow feature map, W is the general term of the allocated weight matrix of the output channel of the compressed shallow feature map, wherein W1 and W2 are the weight matrices allocated by the first two full connections, and the weight matrices are all three-dimensional matrices; σ represents sigmoid function, δ represents Relu activation function, before sigmoid function, W1 and z are multiplied by matrix operation to realize the first full connection; after the Relu layer processing, W2 is multiplied to complete the second full connection;

[0107] Two full connections are used to reduce the number of channels and the amount of calculation, the dimension of W1 is 1*1*C / r, r is a scaling parameter, the dimension of W1 multiplied by z is 1*1*C / r; after multiplication, the dimension is not changed after the Relu layer processing; then multiplied by W2, the second full connection is completed, the dimension of W2 is 1*1*C*C / r, and the output dimension of the second full connection is 1*1*C;

[0108] S33, the neural network multiplies the input shallow feature map and the assigned weight value of the feature channel to obtain a shallow feature map after feature enhancement;

[0109] The expression of the shallow feature map after feature enhancement is:

[0110]

[0111] Wherein, is the shallow feature map after feature enhancement, F scale is a feature enhancement operation, u c , s c respectively, the shallow feature map input through the channel c and the assigned weight value of the feature channel c.

[0112] In the calculation process of each shallow feature quality score in the embodiment S4:

[0113] S41, the shallow feature maps of the same layer of the reference image and the distorted image are converted into one-dimensional feature vectors and normalized to obtain normalized probability distribution feature maps;

[0114] S42, based on the two normalized probability distribution feature maps, the WSD value between the distorted image and the reference image is calculated using the Wasserstein distance;

[0115] S43, the calculated WSD value is mapped to the image quality evaluation score, which is used to evaluate the degradation degree of the distorted image;

[0116] In S41, the expression of the normalized probability distribution feature map is:

[0117]

[0118] Wherein, P is the normalized probability distribution feature map, f is the shallow feature map after feature enhancement, C is the feature enhancement channel, H and W are the length and width of the shallow feature map after feature enhancement, f i represents a single feature element in the shallow feature map after feature enhancement, i takes a value ranging from 1 to CxHxW;

[0119] In S42, the WSD value between the shallow feature image pairs of the same layer of the distorted image and the reference image is calculated using the Wasserstein distance, and the expression is:

[0120]

[0121] wherein X and Y represent the normalized probability distribution features of the reference image and the distorted image respectively, represent the cumulative probability distribution functions of X and Y respectively, which are calculated based on the normalized probability distribution P of the reference image and the distorted image; WSD is the image degradation distance between the two normalized probability distribution features of X and Y.

[0122] In the embodiment, S5 comprises:

[0123] S51, performing self-attention enhancement on the deep feature maps of the last two layers of the reference image and the distorted image to obtain enhanced deep feature maps;

[0124] S52, calculating the perceptual difference of the image degradation degree of the reference image and the distorted image according to the enhanced deep feature maps of the reference image and the distorted image;

[0125] S53, inputting the perceptual difference of the two layers into the attention aggregation module for batch normalization processing to obtain aggregated deep feature maps;

[0126] S54, outputting the aggregated deep feature maps to the splicing module for preliminary processing, dropout layer processing and connection input to obtain finally spliced deep feature maps;

[0127] S55, inputting the finally spliced deep feature maps into the fusion module for fusion operation to obtain fused deep feature maps;

[0128] S56, obtaining the final deep feature quality score based on the fused deep feature maps using the weighted score and the pixel prediction.

[0129] In the embodiment, in S51, the self-attention enhancement comprises normalization processing, which is used to capture the internal correlation information and enhance the original input deep feature maps;

[0130] The expression of the enhanced deep feature maps is:

[0131]

[0132] wherein Q, K and V represent the query vector, the key vector and the value matrix vector in the self-attention mechanism respectively, and Softmax is a normalization function; d k is the dimension of the key vector, which is used to scale the dot product to avoid the problem of gradient disappearance when the dimension is large;

[0133] In S51, the expression of the deep feature maps after the self-attention enhancement of the i-th layer is:

[0134] f i = Attn(F i W q , F i W k , F i W v ) + F i

[0135] wherein f i is the i-th layer deep feature map after self-attention enhancement, F i is the i-th layer deep feature map before self-attention enhancement, and i is 4 or 5; W q , W k and W v respectively represent the weights of the query vector, the key vector and the value matrix vector in the self-attention enhancement;

[0136] In S52, the formula for calculating the perceptual difference between the i-th layer deep feature map of the reference image and the distorted image after self-attention enhancement is:

[0137]

[0138] wherein, is the perceptual difference between the i-th layer deep feature map of the reference image and the distorted image after self-attention enhancement, are the deep feature maps of the reference image and the distorted image after self-attention enhancement, respectively;

[0139] In S53, the expression of the aggregated deep feature map is:

[0140]

[0141] In S54, the expression of the preliminary processed deep feature map is:

[0142]

[0143] wherein BatchNorm represents batch normalization processing, is the aggregated deep feature map, is the preliminary processed deep feature map;

[0144] The expression of the i-th layer deep feature map after the dropout layer processing is:

[0145]

[0146] wherein Dropout represents the dropout layer operation, is the i-th layer deep feature map after preliminary processing; is the aggregated i-th layer deep feature map, is the i-th layer deep feature map after the discard layer processing;

[0147] The expression of the last spliced deep feature map after input connection is:

[0148]

[0149] wherein, is the last spliced deep feature map after input connection, is the 4th and 5th layer deep feature map after the discard layer processing, respectively;

[0150] In S55, the expression of the fused deep feature map is:

[0151]

[0152] wherein, F fusion is the fused deep feature map, and the fusion process includes 3X3 convolution, Relu activation function, two consecutive 3X3 convolution and Relu function output on the last spliced deep feature map .

[0153] In S56, the expression of the last deep feature quality score is:

[0154] d 45 =Conv(F fusion )⊙Sigmoid(Conv(F fusion ))

[0155] wherein, d 45 is the last deep feature quality score, F fusion is the fused deep feature map, which is input into the double-branch prediction module, one branch is used for pixel prediction, and the other branch is used for weighted score prediction; and ⊙ represents dot product.

[0156] In the embodiment S6, the expression of the final deep and shallow layer full reference image quality score after the integration of the deep learning model is:

[0157] D=α1d1+α2d2+α3d3+α 45 d 45

[0158] wherein, α1, α2, α3, α 45 are four learnable parameters in the deep learning model, respectively, indicating the key degree of different layers to the final quality evaluation score, d1, d2, d3 are the quality evaluation scores of the first three layer shallow feature maps, respectively, and d 45 is the last deep feature quality score.

[0159] In the embodiment, the four learnable parameters of the deep learning model are respectively the key degrees of the shallow feature quality scores of the first three layers of feature maps and the deep feature quality score of the last layer of feature maps in the full reference image quality evaluation.

[0160] In the embodiment, the four learnable parameters of the deep learning model can be set in advance according to needs.

[0161] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0162] The above description of the disclosed embodiments enables a person skilled in the art to implement or use the present application. Various modifications to the embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A deep and shallow layer full-reference image quality assessment method based on visual perception, characterized in that, The method comprises the following steps: S1, obtaining a pair of reference images and distorted images with unified sizes; S2, extracting five layers of feature maps based on the reference images and the distorted images, wherein the first three layers of feature maps are shallow feature maps, and the last two layers of feature maps are deep feature maps; S3, performing feature enhancement on the first three layers of shallow feature maps respectively; S4, using the Wasserstein distance to measure the difference between the reference images and the distorted images in the same layer of shallow feature maps based on the first three layers of shallow feature maps after feature enhancement, and calculating three shallow feature quality scores; S5, performing self-attention enhancement on the last two layers of deep feature maps, and calculating the image perception difference between the reference images and the distorted images in the same layer of deep feature maps, so as to measure the degradation degree of the deep feature maps in the same layer; the image perception difference between the two layers of deep feature maps is processed through an attention aggregation module, a splicing module and a fusion module in sequence, and a fused deep feature quality score is obtained by using a weighted score and pixel prediction; S6, a deep learning model integrates the three shallow feature quality scores and one deep feature quality score through four learnable parameters to obtain a final deep and shallow full-reference image quality score; In the step S3, the first three layers of shallow feature maps of the reference images and the distorted images are enhanced by using the SENet, and the process is as follows: S31, the SENet performs compression operation on the input shallow feature map along the spatial dimension to obtain a corresponding compressed vector, and the expression is as follows: wherein c denotes a specific feature channel, u c a shallow feature map inputted for the feature channel c, F sq denotes a compression operation; z c a vector after the shallow feature map inputted for the feature channel c is compressed, which has a dimension of 1*1*C, C is a real number with a global receptive field, which ensures that the dimension of the output matches the number of feature channels of the input; H and W are the length and width of the inputted shallow feature map, respectively; S32, the Excitation operation is used to identify the weight value allocated to the output channel of the compressed shallow feature map, and the expression is as follows: s = F ex (z,W) = σ(g(z,W)) = σ(W2δ(W1z)) wherein s is the weight assigned to the compressed shallow feature map output channel, F ex represents the Excitation operation, z is the compressed vector of the shallow feature map, W is the collective term for the weight matrix assigned to the compressed shallow feature map output channel, wherein W1 and W2 are the weight matrices assigned by the first two full connections, both of which are three-dimensional matrices; σ represents the sigmoid function, and δ represents the Relu activation function. Before the sigmoid function, W1 is multiplied by z in a matrix multiplication operation to realize a full connection. After being processed by the Relu layer, W2 is multiplied to complete the second full connection, which is used to reduce the number of channels and the amount of calculation. The dimension of W1 is 1*1*C / r, r is a scaling parameter, and the dimension of W1 multiplied by z is 1*1*C / r; after multiplication, the dimension of the processed result through the Relu layer is unchanged; after multiplication with W2, the dimension of W2 after the second full connection is 1*1*C*C / r, and the output dimension after the second full connection is 1*1*C; S33, the neural network multiplies the input shallow feature map with the allocated weight value of the feature channel to obtain the shallow feature map after feature enhancement; The expression of the shallow feature map after feature enhancement is as follows: wherein, is the feature-enhanced shallow feature map, F scale is the feature-enhancing operation, u c , s c are the shallow feature map and the assigned weight of the feature channel c, respectively, input through the channel c.

2. The visual perception based deep and shallow layer full reference image quality assessment method of claim 1, wherein, In S1, the image sizes of the obtained reference images and the distorted images are unified to 224x224 pixels, and the image size unification manner is to unify the image size by using an image processing software or library.

3. The visual perception based deep and shallow layer full reference image quality assessment method of claim 1, wherein, In S2, the following steps are included: S21, inputting the reference images and the distorted images with unified sizes into a visual geometry group network model VGG-16; S22, the visual geometry group network model VGG-16 extracts the first three layers of shallow feature maps of the reference images and the distorted images based on the texture, color and shape of the images; S23, the visual geometry group network model VGG-16 extracts the last two layers of deep feature maps of the reference images and the distorted images based on the high-level semantic information of the images.

4. The visual perception based deep and shallow layer full reference image quality assessment method of claim 1, wherein, In S4, the calculation process of each shallow feature quality score is as follows: S41, converting the shallow feature maps of the reference images and the distorted images in the same layer into one-dimensional feature vectors and performing normalization processing to obtain normalized probability distribution feature maps; S42, calculate the WSD value between the distorted image and the reference image based on the two normalized probability distribution feature maps using the Wasserstein distance; S43, map the calculated WSD value to the image quality evaluation score, which is used to evaluate the degradation degree of the distorted image.

5. The visual perception based deep and shallow layer full reference image quality assessment method of claim 4, wherein, In S41, the normalized probability distribution feature map is expressed as: wherein P is a normalized probability distribution feature map, f is a shallow feature map after feature enhancement, C is a feature enhancement channel, H and W are the length and width of the shallow feature map after feature enhancement, f i represents a single feature element in the shallow feature map after feature enhancement, and i has a value range of 1 to CxHxW; In S42, the WSD value between the distorted image and the reference image is calculated using the Wasserstein distance, and the expression is: wherein X and Y represent the normalized probability distribution feature maps of the reference image and the distorted image, respectively, respectively represent the cumulative probability distribution functions of X and Y, which are calculated based on the normalized probability distributions P of the reference image and the distorted image, respectively; and WSD represents the image degradation distance between the two normalized probability distribution feature maps of X and Y.

6. The visual perception based deep and shallow layer full reference image quality assessment method of claim 1, wherein S5 It includes: S51, the deep feature maps of the last two layers of the reference image and the distorted image are enhanced by self-attention, and the enhanced deep feature maps are obtained; S52, calculate the perceptual difference between the reference image and the distorted image based on the enhanced deep feature maps; S53, input the perceptual difference of the two layers into the attention aggregation module for batch normalization processing to obtain the aggregated deep feature map; S54, output the aggregated deep feature map to the splicing module for preliminary processing, discard layer processing and connection input to obtain the finally spliced deep feature map; S55, input the finally spliced deep feature map into the fusion module for fusion operation to obtain the fused deep feature map; S56, based on the fused deep feature map, the final deep feature quality score is obtained by using the weighted score and pixel prediction.

7. The visual perception based deep and shallow layer full reference image quality assessment method of claim 6, wherein, In S51, the self-attention enhancement includes normalization processing, which is used to capture the internal correlation information of the original deep feature map and enhance the feature map; The expression of the enhanced deep feature map is: wherein Q, K, V represent query vector, key vector and value matrix vector in the self-attention mechanism respectively, and Softmax is a normalization function; d k is the dimension of the key vector.

8. The visual perception based deep and shallow layer full reference image quality assessment method of claim 6, wherein, In S51, the expression of the i-th layer of the self-attention enhanced deep feature map is: f i = Attn(F i W q , F i W k , F i W v ) + F i wherein f i is the deep feature map of the i-th layer after self-attention enhancement, F i is the deep feature map of the i-th layer before self-attention enhancement, i is 4 or 5; W q , W k and W v respectively represent the weights of the query vector, the key vector and the value matrix vector in the self-attention enhancement. In S52, the perceptual difference between the i-th layer of the self-attention enhanced deep feature map of the reference image and the distorted image is calculated as: wherein, is the perceptual difference between the i-th layer of deep feature maps of the reference image and the distorted image after feature enhancement, are the deep feature maps of the reference image and the distorted image after self-attention enhancement, respectively; In S53, the expression of the aggregated deep feature map is: In S54, the expression of the preliminary processed deep feature map is: wherein, BatchNorm represents batch normalization processing, is the deep feature map after the aggregation, is the deep feature map after the preliminary processing. The expression of the i-th layer of the deep feature map after discard layer processing is: wherein Dropout represents a dropout layer operation, is the i-th layer deep feature map after preliminary processing; is the i-th layer deep feature map after aggregation, is the i-th layer deep feature map after dropout layer processing; The expression of the finally spliced deep feature map after input connection is: wherein, is the last spliced depth feature map after connection of the input, respectively, are the 4th and 5th layer depth feature maps after the discard layer processing; In S55, the expression of the fused deep feature map is: Wherein, F fusion is the fused deep feature map, and the fusion process includes performing 3X3 convolution, a Relu activation function, two consecutive 3X3 convolutions, and a Relu function output on the last spliced deep feature map performing 3X3 convolution, a Relu activation function, two consecutive 3X3 convolutions, and a Relu function output In S56, the expression of the final deep feature quality score is: d 45 = Conv(F fusion ) Θ Sigmoid(Conv(F fusion )) wherein d 45 is the final deep feature quality score, F fusion is the fused deep feature map, which is input into a dual-branch prediction module, one branch for pixel prediction and the other branch for weighted score prediction; and ⊙ denotes dot product.

9. The visual perception based deep and shallow layer full reference image quality assessment method of claim 1, wherein, In S6, the expression of the final deep and shallow layer full reference image quality score of the integrated deep learning model is: D = α1d1+ α2d2+ α3d3+ α 45 d 45 wherein, a1, a2, a3, a 45 are four learnable parameters in the deep learning model, representing the key degree of different layers to the final quality evaluation score, d1, d2, d3 are the shallow feature quality scores of the first three layers of feature maps, respectively, and d 45 is the last deep feature quality score.

Images