A no-reference image quality evaluation method based on graph convolution and multi-scale features

Abstract

The application provides a no-reference image quality evaluation method based on graph convolution and multi-scale features, which models the complex relationship of image distortion through a graph convolution network. In the pre-training process, the feature extraction network Resnet50 is trained by using the double-layer gradient optimization strategy of meta-learning, and the local and global multi-scale features are fused at the same time, so that the long-distance spatial and inter-channel dependence relationship around each spatial position is adaptively constructed, and a spatial pyramid pooling layer is added, so that the network can accept images of any ratio as input, and the evaluation process is similar to human perception; secondly, the graph representation of distortion-related information is established through the graph convolution network, the distortion characteristics are integrated, and the discrimination of the features is improved; finally, the pre-trained model verifies its effectiveness on the target data set. The method can effectively obtain the global distortion relationship, enhance the feature representation of the distortion, and realize good quality prediction performance for synthetic distortion and real distortion.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and particularly relates to a no-reference image quality evaluation method based on graph convolution and multi-scale features. BACKGROUND

[0002] With the development and popularization of electronic information technology and intelligent devices, multimedia digital images as the carrier of information transmission have been widely used in medical health, education and life, weather forecast and the like. However, images will be distorted to different degrees in the process of transmission, compression and storage, thereby leading to the decline of image quality. Therefore, the objective image quality evaluation (IQA) method capable of automatically evaluating image quality has been a long-term research topic in the field of image processing and has a wide application in the fields of image compression, image retrieval, image fusion and virtual reality.

[0003] In the scope of IQA, no-reference image quality evaluation (NR-IQA) has attracted much attention because there is often no reference in many real-world applications. Meanwhile, learning-based NR-IQA methods perform well due to the powerful fitting ability of deep convolutional neural networks (DCNNs). They automatically capture deep features to represent degradation and are widely used in the NR-IQA task. However, DCNN uses end-to-end training to establish the complex relationship between model parameters and image distortion, and it is easy to ignore the detailed information existing in the distorted image, so the DCNN-based method also has the following problems: how to obtain the connection between the local and global of the image, how to solve the problem caused by the preprocessed image, how to establish an effective distortion representation of the distorted image and the like. SUMMARY

[0004] The technical problem to be solved by the present application is at least one of the following: how to obtain the connection between the local and global of the image, how to solve the problem caused by the preprocessed image, and how to establish an effective distortion representation of the distorted image.

[0005] In order to achieve the above object, the present application proposes a new image quality evaluation method based on graph convolution and multi-scale features, which pre-trains model parameters based on a meta-learning framework to further improve the generalization ability to distorted data. The present application can accept images of any ratio, avoiding changes in input image quality caused by preprocessing methods. In order to enhance the connection between local and global, the present application designs an adaptive fusion multi-scale feature module, which adaptively constructs a larger receptive field space and inter-channel correlation for each spatial position, uses long-distance features to guide the feature change of the original features, and fuses more rich information to enhance feature representation. Then, the present application uses a graph convolution network to model the distortion-related factors affecting the perceived quality to obtain the graph representation of the distorted image and improve the performance of the model. The final model test results show that the model proposed by the present application can effectively obtain the global distortion relationship, enhance the feature representation of the distortion, and realize good quality prediction performance for synthetic distortion and real distortion.

[0006] The present application provides a no-reference image quality evaluation method based on graph convolution and multi-scale features, comprising the following steps:

[0007] A no-reference image quality evaluation model is built, which includes a pre-training stage and a fine-tuning stage;

[0008] A no-reference image is obtained, and a training set is constructed;

[0009] In the pre-training stage, the training set is input into the model, and a meta-learning method is used to train the Resnet backbone network through a double-layer gradient optimization strategy;

[0010] At the same time, an adaptive fusion multi-scale feature module is introduced to adaptively fuse image local and global features to obtain multi-scale features, construct global spatial and channel dependency, and then train a meta-model containing specific distortion knowledge;

[0011] A spatial pyramid pooling layer is added after the meta-model containing specific distortion knowledge, which is used to receive images of any size;

[0012] The multi-scale features are subjected to global pooling to obtain a feature set, which is then fed into a node construction module and an edge construction module to obtain a graph representation of a specific distortion type;

[0013] The nodes and edges of the graph representation of the specific distortion type are input into a distortion type discrimination module to obtain a triple loss, and the contrast relationship between different distortions is learned;

[0014] The mean square error loss is obtained through a distortion level discrimination module, and the overall model is optimized by weighting the triple loss to obtain prior knowledge for constructing a distortion graph representation;

[0015] In the fine-tuning stage, the distorted image is extracted by the pre-trained feature extraction network to obtain the distorted feature;

[0016] The node construction module and the edge construction module with prior knowledge are used to quickly obtain a good representation of the distorted feature;

[0017] The good representation of the distorted feature is mapped through the full connection layer to obtain the corresponding quality score.

[0018] Further, when training Resnet to adapt to unknown distortion by double-layer gradient optimization, an adaptive fusion multi-scale feature module is introduced, the convolution block is divided into multiple parts, the feature change of another part of the convolution block is calibrated through the transformation of a convolution block, and the global space and inter-channel dependency relationship is adaptively constructed.

[0019] Further, the adaptive fusion of image local and global features obtains multi-scale features and constructs global space and inter-channel dependency relationship, including:

[0020] For a given set of shape C, H, W feature layers, C, H, W represent the number of channels, height and width respectively, which are divided into 3 convolution blocks M1, M2, M3;

[0021] The input feature X is divided into two parts by channel segmentation operation X local ,X global represent local features and global features respectively;

[0022] X global is calibrated by M1, M2, given input X global , using average pooling with size R×R and stride R to expand the receptive field:

[0023] P1=AVGPool R (X global )

[0024] P1 is feature transformed by M1:

[0025] X' global =UP(P1×M1)

[0026] where UP(·) is a bilinear interpolation operator, which increases the calibration operation after feature transformation; X global is enhanced by M2:

[0027] Y'=(X global ×M2)·σ(X global +X' global )

[0028] where σ is an activation function, and · represents dot multiplication of features;

[0029] Using X' global As residual factors, global distortion information is captured, and the captured global distortion information is combined with the original feature X local to obtain the output vector Y:

[0030]

[0031] In the formula, indicates that the features are added correspondingly.

[0032] Further, the feature set is delivered to the node construction module and the edge construction module, and the step of obtaining the graph representation of a specific distortion type includes:

[0033] The node construction module is composed of multiple fully connected layers, and the specific expression of inputting the feature set into the node construction module is as follows:

[0034]

[0035] Wherein, f k,i indicates the feature vector of each image, indicates the node feature dimension, N indicates the number of distorted images, and F N (f k,i ; θ) indicates the node construction network, θ indicates the network parameters of F N , v k,i indicates the node generated by the i-th sample in the distortion type k, V k indicates the node feature set of the distortion graph representation.

[0036] Further, the feature set is delivered to the node construction module and the edge construction module, and the step of obtaining the graph representation of a specific distortion type includes:

[0037] The value of the edge feature is initialized by the dot product between nodes, and is expressed as indicates the node feature dimension, i and j indicate the relationship between the i-th sample and the j-th sample;

[0038] After initializing the edge feature, the edge construction module composed of the graph convolution network further optimizes the internal structure of the graph representation, and the specific process is as follows:

[0039] The initialized edge feature and the corresponding adjacency matrix are input into the edge construction module, and the value of the edge feature after optimization by the l-layer graph convolution network can be calculated as:

[0040]

[0041] Where σ represents the ReLu operation, represents the three-dimensional adjacency matrix, I represents the identity matrix, represents the output of the l-th layer graph convolutional network, is the weight matrix of the lth layer of the graph convolutional network;

[0042] The edge feature set represented by the distortion graph can be expressed as:

[0043]

[0044] in, represents the edge feature dimension, e k,i,j Indicates the relationship between the i-th sample and the j-th sample, E k A set of edge features representing the distorted graph.

[0045] Furthermore, after the distorted image is represented as a graph structure, the global information from the node feature set and the relationship between samples from the edge feature set are aggregated through the distortion type discrimination module, and the specific expression of the triplet loss function is obtained as follows:

[0046]

[0047]

[0048] Among them, l represents the number of layers of the distortion type discrimination module composed of the graph convolutional network, Represents node V k The lth and l+1th layers, σ represents the ReLu operation, T represents the distortion type discrimination module, Indicates E k Perform average pooling transformation to obtain node V k The two-dimensional adjacency matrix of is the weight matrix of the lth layer of the graph convolutional network, y code represents the output vector of the distortion type discrimination module, d represents the Euclidean distance, Represents and y code Samples belonging to the same type are positive samples, and Represents and y code Different types of samples represent negative samples, and margin represents the threshold that separates positive and negative samples when comparing. represents the triplet loss function.

[0049] Furthermore, the edge feature E k By averaging the pooled input, aggregating information from the current node and the remaining adjacent nodes, and using the mean square error loss function to train the distortion level discrimination module, the specific expression is:

[0050]

[0051]

[0052] Among them, y i and y i 'represent the predicted and actual distortion levels respectively, i represents the i-th sample, represents the mean square error function, Represents the triple loss function, which is the function of the two loss functions by the hyperparameter λ and Weighted to obtain the loss function of the overall model

[0053] Furthermore, in the fine-tuning phase, the outputs of the node building module and the edge building module, which have learned rich prior knowledge, are spliced ​​together and then input into the fully connected layer to predict the final quality score. The mean square error function of the training model is specifically:

[0054]

[0055] Among them, N f represents the mini-batch size for training, y i and y i 'represent the predicted and actual distortion levels respectively, and i represents the i-th sample.

[0056] Furthermore, the backbone network is a Resnet50 feature extraction network.

[0057] In addition, in order to achieve the above-mentioned purpose, the present invention also provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for no-reference image quality evaluation based on graph convolution and multi-scale features are implemented.

[0058] The application provides a no-reference image quality evaluation method based on graph convolution and multi-scale features, aims to solve the problems of other quality evaluation methods based on distortion representation and the deficiencies of existing convolution networks, introduces a meta-learning training method in training, optimizes and trains a Resnet50 backbone network through double-layer gradient, simultaneously introduces a self-adaptive fusion multi-scale feature module, uses enhanced convolution operation to expand the receptive field of convolution, and calibrates the relationship between global contexts by fusing local and global features. Then, after obtaining the meta-model, a spatial pyramid pooling is added to realize processing of images of any size, subsequently, a node construction module and an edge construction module are used to model the graph representation structure of a specific distortion type, and the model is optimized through a distortion type discrimination module and a distortion level discrimination module, and the pre-trained model uses rich prior knowledge to predict any distorted image. The whole model can effectively obtain the global distortion relationship, enhance the feature representation of distortion, and realize good quality prediction performance on synthetic distortion and real distortion. BRIEF DESCRIPTION OF DRAWINGS

[0059] The application will be further described below in combination with the drawings and examples, and the drawings are as follows:

[0060] Figure 1 An execution flowchart of the no-reference image quality evaluation method based on graph convolution and multi-scale features provided by the embodiment of the application is provided.

[0061] Figure 2 A no-reference image quality evaluation model structure schematic diagram provided by the embodiment of the application is provided.

[0062] Figure 3 A self-adaptive fusion multi-scale feature module structure schematic diagram provided by the embodiment of the application is provided.

[0063] Figure 4 A graph representation optimization schematic diagram provided by the embodiment of the application is provided.

[0064] Figure 5 A feature extraction network feature extraction visualization schematic diagram provided by the embodiment of the application is provided.

[0065] Figure 6 A feature map comparison result of a Layer1 layer in the pre-trained feature extraction network provided by the embodiment of the application is provided. DETAILED DESCRIPTION

[0066] Various exemplary embodiments of the application will be described in detail below in combination with the drawings. It should be noted that the relative arrangement, numerical expression and values of components and steps set forth in these embodiments do not limit the scope of the application unless otherwise specifically stated.

[0067] It should be understood that the dimensions of the various parts shown in the drawings are shown for purposes of illustration only and are not intended to be limiting.

[0068] The following description of at least one exemplary embodiment is merely exemplary in nature and is in no way intended to limit the application or its application and uses.

[0069] For the purpose of this disclosure, the term "coupled" means the direct or indirect connection between or among individuals, devices, components, or modules, for example, means that each is individually coupled, or two or more are coupled together either directly or indirectly.

[0070] Techniques, methods, and devices known to those of ordinary skill in the relevant art can not be discussed in detail herein. However, where appropriate, such techniques, methods, and devices can be considered part of the present disclosure.

[0071] In all of the examples shown and discussed herein, any specific values should be interpreted as merely illustrative and not as a limitation. Thus, other examples of the exemplary embodiments can have different values.

[0072] It should be noted that like reference numerals and letters refer to like items in the following drawings, and thus, once an item is defined in one drawing, it is not necessary to discuss it further in subsequent drawings.

[0073] Referring to Figure 1 and Figure 2 , the embodiment of the present application provides a no-reference image quality evaluation method based on graph convolution and multi-scale features, comprising the following steps:

[0074] S1: a no-reference image quality evaluation model is built, which includes a pre-training stage and a fine-tuning stage;

[0075] S2: a no-reference image is obtained, and a training set is constructed;

[0076] S3: in the pre-training stage, the training set is input into the model, and a meta-learning method is used to train the Resnet backbone network through a double-layer gradient optimization strategy;

[0077] S4: a self-adaptive fusion multi-scale feature module is introduced at the same time, the local and global features of the image are adaptively fused to obtain multi-scale features, the dependence relationship between the global space and the channel is constructed, and then a meta-model containing specific distortion knowledge is trained;

[0078] Specifically, referring to Figure 3, because human vision not only perceives the overall semantic information of the image from a global perspective when perceiving distorted images, but also perceives the local details of interest in the image. Therefore, considering the locality problem of the convolutional network, an adaptive fusion multi-scale feature module is introduced to divide the convolution block into multiple parts. The feature changes of another part of the convolution block are calibrated by the transformation of one convolution block, thereby effectively expanding the receptive field of each spatial position and adaptively constructing the dependency relationship between the global space and channels. The specific steps of the adaptive fusion multi-scale feature module introduced by the present invention are as follows: for a set of feature layers with a given shape of C, H, W, C, H, W represent the number of channels, height and width respectively, divide it into three convolution blocks M1, M2, and M3;

[0079] The input feature X is divided into Two parts, X local ,X global Represent local features and global features respectively;

[0080] Use M1, M2 to analyze the global feature X global Perform calibration operation, given input X global , using average pooling of size R×R and stride R to expand the receptive field:

[0081] P1=AVGPool R (X global )

[0082] Perform feature transformation on P1 through M1:

[0083] X' global =UP(P1×M1)

[0084] Where UP(·) is a bilinear interpolation operator. After feature transformation, a calibration operation is added. global To perform channel enhancement:

[0085] Y'=(X global ×M2)·σ(X global +X' global )

[0086] Where σ is the activation function and · represents the dot product of the features;

[0087] Use X' global As a residual factor, it captures the global distortion information, and the captured global distortion information is compared with the original local feature X through M3. local Fusion is performed to obtain the output vector Y:

[0088]

[0089] Where, The feature pair is represented by addition.

[0090] S5, add a spatial pyramid pooling layer after the meta-model containing specific distortion knowledge, for receiving images of arbitrary size.

[0091] S6, obtain a feature set by globally pooling the multi-scale features, and then deliver the feature set to a node construction module and an edge construction module to obtain a graph representation of a specific distortion type.

[0092] Specifically, referring to Figure 2 , most of the existing methods regard different distortion types as a flat model, without simulating the relationship between distortion types and degrees, or considering the influence of image content on perceived quality, the present application models the relationship between distortion types and distortion levels of an image as a hierarchical model, and represents each specific distortion type as a graph structure. The graph representation of distortion type k can be defined as G k =(V k ,E k ), V represents a node, and E represents an edge. Extract N distorted images to obtain the corresponding feature set F k ={f i |i=1,2,...,N} in the model, and then input the node construction module and the edge construction module, the specific implementation process is as follows:

[0093] The node construction module is composed of multiple fully connected layers, and the specific expression for inputting the feature set into the node construction module is as follows:

[0094]

[0095] wherein f k,i represents the feature vector of each image, represents the node feature dimension, N represents the number of distorted images, and F N (f k,i ; θ) represents the node construction network, θ represents the network parameters of F N , v k,i represents the node generated by the i-th sample in distortion type k, V k represents the node feature set of the distorted graph representation.

[0096] The value of edge feature is initialized by the dot product between nodes, and is represented as i, j represent the relationship between the i-th sample and the j-th sample;

[0097] After initializing the edge feature, the edge construction module composed of a graph convolution network is input to further optimize the internal structure of the graph representation, and the specific process is as follows:

[0098] The initialized edge feature and the corresponding adjacency matrix As the input of the edge building module, the value of the edge feature obtained after optimization by the l-layer graph convolutional network can be calculated as:

[0099]

[0100] Where σ represents the ReLu operation, represents the three-dimensional adjacency matrix, I represents the identity matrix, Represents the output of the l-th layer GCN (graph convolutional network), is the weight matrix of the lth layer of the graph convolutional network;

[0101] The edge feature set represented by the distortion graph can be expressed as:

[0102]

[0103] in, represents the edge feature dimension, e k,i,j Indicates the relationship between the i-th sample and the j-th sample, E k A set of edge features representing the distorted graph.

[0104] S7. Input the nodes and edges of the graph representing a specific distortion type into the distortion type discrimination module to obtain the triplet loss and learn the comparative relationship between different distortions.

[0105] Specifically, see Figure 4 In order to learn the distortion information of different distortion types and the comparative relationship between various distortions to achieve better generalization performance, this paper proposes a distortion type discrimination module. It uses a multi-layer graph convolutional network to aggregate global information from the node feature set and the relationship between samples from the edge feature set, and uses a triplet loss function to train the model. The specific implementation process is as follows:

[0106]

[0107]

[0108] Among them, l represents the number of layers of the distortion type discrimination module composed of the graph convolutional network, Represents node V k The lth and l+1th layers, σ represents the ReLu operation, T represents the distortion type discrimination module, Indicates E k Perform average pooling transformation to obtain node V k The two-dimensional adjacency matrix of is the weight matrix of the lth layer of the graph convolutional network, y codeThe output vector of the distortion type discrimination module, d represents the Euclidean distance, represents and y code The samples belonging to the same type represent positive samples, and represents and y code The samples of different types represent negative samples, and margin represents the threshold value for separating positive and negative samples during comparison, represents the triplet loss function, where, Not only can the more subtle differences between distortion types be learned, but the contrast relationship between distortion types can also be learned, avoiding overfitting of the network to the distortion types in the training set.

[0109] S8, obtain the mean square error loss through the distortion level discrimination module, and weight the triplet loss to optimize the overall model, and obtain the prior knowledge for constructing the distortion graph representation.

[0110] Specifically, referring to Figure 4 , in order to predict the degree of distortion, while considering the uncertainty caused by image content, the distortion level discrimination module is proposed, which predicts the degree of distortion by analyzing the information contained in the node features and the comparison between nodes, and inputs the node features V k , and the edge features E k are inputted through average pooling, and are represented as E' k =[∑ j e k,i,j ] / N, the information from the current node and the remaining adjacent nodes is aggregated through the two branches, and the mean square error loss function is obtained to train the distortion level discrimination module, and specifically:

[0111]

[0112]

[0113] Wherein, y i and y i ' represent the predicted and true distortion degree respectively, i represents the i-th sample, represents the mean square error function, and the two loss functions and are weighted by the hyperparameter λ to obtain the loss function of the overall model

[0114] S9, in the fine-tuning stage, the distorted image is extracted through the pre-trained feature extraction network.

[0115] S10, the node construction module and the edge construction module with prior knowledge are used to quickly obtain a good representation of the distortion feature.

[0116] S11, the good representation of the obtained distortion feature is mapped to the corresponding quality score through a fully connected layer.

[0117] Specifically, referring to Figure 2 , the outputs of the node construction module and the edge construction module learned from the rich prior knowledge are spliced and then input into a fully connected layer to predict the final perceptual quality score, and the overall model is trained using a mean square error function, specifically:

[0118]

[0119] Where, N f represents the size of the small batch of training.

[0120] In the embodiments of the present application, the beneficial effects of the present application are also verified through experiments. First, the experimental settings, implementation details and data set selection, etc. are described in the experimental scheme. Then, the overall data set performance evaluation and single distortion data set performance evaluation with other methods are carried out. Finally, the ablation experiment is carried out to verify the effectiveness of each module in the present application.

[0121] The training process of the method of the present application includes two stages: (1) in the pre-training stage, the meta-model network fused with multi-scale features and the graph representation construction network are trained on the synthetic data set; (2) in the fine-tuning stage, the overall network is trained on the target data set and the parameters are fine-tuned.

[0122] During pre-training, the meta-model pre-trained under the meta-learning framework is used to construct the feature extraction network, and then the overall model network is trained using the Kadid-10k data set. The hyperparameter λ of the loss function is set to 0.25. The threshold value margin of the triplet loss is set to 0.1. The network parameters are trained 40000 times using the Adam optimization algorithm, the mini-batch size is 64, and the learning rate is set to 1e-7. The dimension size of C V in the node construction network is set to 256, and the size of C E in the edge construction network is set to 64.

[0123] During fine-tuning, two datasets with real distortions (KonIQ-10k and LIVEC) and two datasets with synthetic distortions (LIVE and CSIQ) are selected for the target dataset. During fine-tuning, the dataset is split into training and test sets. Specifically, 80% of the images randomly selected from the real distortion datasets KonIQ-10k and LIVEC are used to train all the models, and the remaining 20% of the images are tested. In the LIVE and CSIQ datasets, the reference images are segmented using an 8:2 training-test ratio, which helps to make the image content independent between the training set and the test set. All results are obtained by training and testing 10 random partition operations on a specific target dataset, and the average results are reported. The present application uses the Adam optimization algorithm to fine-tune the target IQA (Image Quality Assessment) task 20 times, with a mini-batch size of 32 and a learning rate set to 5e-6. The present application uses the evaluation indicators SROCC and PLCC to measure the predicted results.

[0124] To evaluate the prediction accuracy of the method of the present application, the present application selects 11 representative IQA methods for experimental comparison, including methods based on manually extracted features, synthetic distortion IQA methods based on deep learning, and real distortion IQA methods based on deep learning. The experimental results are shown in Tables 1 and 2, and the best results are highlighted in bold.

[0125] Table 1 SROCC results of different NR-IQA methods

[0126]

[0127] From Tables 1 and 2, it can be observed that the method proposed by the present application achieves SROCC results of 0.849, 0.910, 0.942, and 0.976 and PLCC results of 0.878, 0.926, 0.960, and 0.976 on the LIVEC, Koniq-10k, CSIQ, and LIVE datasets, respectively. The average results of the present application model on the four datasets are all optimal. For the Koniq-10k and LIVE datasets, the method proposed by the present application achieves the best prediction accuracy in SROCC and PLCC. The present application will be discussed from the following three aspects:

[0128] (1) First, the model of the present application is superior to the method based on manual production, whether on synthetic distortion or real distortion dataset. More specifically, the model of the present application exceeds the HOSA method with close performance by about 31.63% on SROCC index on LIVEC, by about 35.82% on Koniq-10k, by about 25.43% on CSIQ, and by about 3.28% on LIVE. While the model of the present application exceeds the HOSA method with close performance by about 29.50% on PLCC index on LIVEC, by about 29.69% on Koniq-10k, by about 16.50% on CSIQ, and by about 3.06% on LIVE. It can be seen from the experimental results that the way of constructing distortion representation by relying on graph convolution network is more sensitive to distortion related factors than traditional methods, so as to more accurately capture key information.

[0129] (2) Compared with the synthetic IQA method based on deep learning, the model of the present application still leads on two real datasets Koniq-10k and LIVE. For SROCC and PLCC indexes, the model of the present application improves by about 15.67% and 18.97% on LIVEC compared with the CaHDC method with more accurate prediction, and exceeds the CaHDC method by about 11.11% and 11.03% on Koniq-10k. It can be seen that although the method of the present application does not use real distortion data in the pre-training stage, it can still perform well on real distortion dataset. This shows that the pre-trained model has the potential to complete the IQA task and can transfer the learned distortion experience to other distortion domains. On the synthetic distortion dataset, the model of the present application also surpasses the method designed specifically for synthetic distortion on LIVE.

[0130] (3) Compared with the real IQA method based on deep learning, the method of the present application achieves the highest prediction accuracy on LIVE and Koniq-10k. On CSIQ, the DBCNN method has better prediction accuracy than the method of the present application on SROCC, while the method of the present application has equivalent performance on PLCC. The main reason is that the bilinear pooling strategy of DBCNN can deal with both synthetic distortion and real distortion, while the method of the present application only uses a small synthetic dataset in the pre-training process. For LIVEC and Koniq-10k datasets, the method of the present application is about 1.50% and 5.11% higher than DBCNN on PLCC. For methods carefully designed for real distortion data, such as HyperIQA and MetaIQA+, the method of the present application also achieves competitive results.

[0131] Through in-depth analysis of the experimental data, it can be concluded that the distortion level representation constructed by the graph nodes and edges of the model has better prediction accuracy than the method based on the traditional manually designed features. Compared with the deep learning based method, the method enhances the convolution to realize multi-scale feature fusion, and further aggregates the global sample information through the GCN, thereby significantly improving the robustness to various distortions.

[0132] Table 2 PLCC results of different NR-IQA methods

[0133]

[0134] The application verifies the prediction effect of the model on specific distortion on two synthetic data sets LIVE and CSIQ with single distortion, and compares with other methods. The experiment includes 5 distortion types in LIVE and 6 distortion types in CSIQ. The SROCC results on the LIVE and CSIQ data sets are shown in Tables 3 and 4.

[0135] Table 3 SROCC results of different distortion types of images on the LIVE data set

[0136]

[0137] Table 4 SROCC results of different distortion types of images on the CSIQ data set

[0138]

[0139] In the SROCC index of Table 3, the model obtains the best evaluation effect for four distortion types in LIVE. It can be seen that for distortion types JPEG, JP2K and GB, the training data set in the construction of the model of the distortion image representation contains rich distortion types, so the method can effectively learn the quality degradation degree of the distortion types. It is worth noting that although the training data set does not contain distortions WN and FF, the model still obtains SROCC results of 0.980 and 0.960. The adaptive multi-scale feature fusion model proposed in the application can effectively obtain distortion information, and retain more image distortion information through the fusion of local region features and global features, and has better robustness.

[0140] In Table 4, the model obtains the best performance for three distortions JPEG, PN and CC of CSIQ, and the performance remains stable in other distortion types, which can be comparable to other deep learning based methods. Similarly, although the distortion WN cannot effectively construct the long distance relationship between the local pixels and the surrounding area, the model still obtains satisfactory results.

[0141] From the experimental results, it can be seen that the SROCC value of the quality evaluation method based on graph convolution and multi-scale feature proposed in the present application is greater than 0.9 in all distortion types, which proves that the pre-trained model can obtain good prior knowledge of distinguishing distortion types and distortion degrees, and can effectively construct good distortion representation for trained or untrained distortion types, gather global information of the image, and help further learn the perception quality.

[0142] The effectiveness of each module in the proposed method is verified by conducting ablation experiments on the LIVEC and LIVE data sets. The SROCC results obtained in the experiments are shown in Table 5. First, for the baseline model (BL), the present application does not perform any pre-training operation on the model, and only uses Resnet50 as the backbone network to crop the distorted image into a specific image block for prediction. The BL model is not trained for distortion types and distortion levels, and has no prior knowledge of various distortion types. As can be observed from the results in the table, the BL model achieves the lowest prediction accuracy. Then, the distortion graph representation construction model based on the graph convolution network is pre-trained, and the distortion type discrimination network and the distortion level discrimination network are used to optimize the model. The model is represented as BL+DGR, which achieves an improvement of 2.89% and 3.15% in SROCC, verifying the effectiveness of the graph representation for modeling distortion-related factors. Then, the adaptive fusion multi-scale feature module (SC) and the meta-learning training process (Meta) are sequentially added to the convolutional network, and the prediction accuracy of the model is further improved. Finally, the spatial pyramid pooling (SPP) module is added, so that the model can accept images of any scale, retaining more image information and quality. The experimental results verify that the features extracted by the method of the present application are more advantageous than the convolutional network used alone, and therefore are more consistent with the perception characteristics of the human visual system.

[0143] Table 5 Ablation study results of LIVEC and LIVE data sets

[0144]

[0145] In order to verify the effectiveness of the feature extraction network designed by the present application, Grad-CAM is used as an attention extraction tool to learn the visualization of the feature map to display the attention of the network. The visualization results of the model of the present application and ResNet50 are shown in Figure 5 As can be observed, the residual network with ordinary convolution cannot capture the entire target area due to the limitation of the receptive field of the convolutional layer. In contrast, the calibrated convolution is not limited to local features, which helps the model of the present application to capture the entire human eye focus area well, thereby generating richer and more discriminative feature representations.

[0146] As shown in Figure 6 , inFigure 6 The feature map extraction results of the Layer1 layer in the feature extraction network are shown. It can be seen from the figure that the improved model of the application can extract more features related to perceptual features compared with the baseline model without adding any improvement module. For example, the model of the application is more sensitive to the edge and texture features of the objects in the image, especially when extracting features of the tomato image, the texture features of the background are also more prominent, and the shape of the cat is effectively outlined, providing more rich information for subsequent construction of the graph structure of the distorted image.

[0147] In addition, the embodiment of the application also provides a storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the steps of the no-reference image quality evaluation method based on graph convolution and multi-scale features, and specifically includes: building a no-reference image quality evaluation model, the model including a pre-training stage and a fine-tuning stage; obtaining a no-reference image and constructing a training set; in the pre-training stage, inputting the training set into the model, using a meta-learning method, and training a Resnet backbone network through a double-layer gradient optimization strategy; simultaneously introducing an adaptive fusion multi-scale feature module, adaptively fusing image local and global features to obtain multi-scale features, constructing a global spatial and channel dependency relationship, and then training a meta-model containing specific distortion knowledge; adding a spatial pyramid pooling layer after the meta-model containing specific distortion knowledge, for receiving images of any size; obtaining a feature set through global pooling of the multi-scale features, and then conveying the feature set to a node construction module and an edge construction module to obtain a graph representation of a specific distortion type; inputting the nodes and edges of the graph representation of the specific distortion type into a distortion type discrimination module to obtain a triplet loss, learning the contrast relationship between different distortions; obtaining a mean square error loss through a distortion level discrimination module, and weighting the triplet loss to optimize the overall model to obtain prior knowledge for constructing a distorted graph representation; in the fine-tuning stage, extracting distortion features through the pre-trained feature extraction network; quickly obtaining a good representation of the distortion features using the node construction module and the edge construction module with prior knowledge; and mapping the obtained good representation of the distortion features through a fully connected layer to obtain a corresponding quality score. The storage medium can achieve the same technical effects when implementing the above steps, and will not be described here.

[0148] The above only describes the preferred embodiments of the application and is not intended to limit the application. Any modifications, equivalent replacements, and improvements made within the spirit and principle of the application should be included in the protection scope of the application.

[0149] It is also to be noted that the terms "comprising", "including", and any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises a... " does not, without further restriction, preclude the existence of additional elements of the same name in the process, method, article, or apparatus.

Claims

1. A method for no-reference image quality assessment based on graph convolution and multi-scale features, characterized in that, The method comprises the following steps: A no-reference image quality evaluation model is built, which includes a pre-training stage and a fine-tuning stage; A no-reference image is obtained, and a training set is constructed; In the pre-training stage, the training set is input into the model, and the Resnet backbone network is trained by using a meta-learning method and a double-layer gradient optimization strategy; Meanwhile, an adaptive multi-scale feature fusion module is introduced to adaptively fuse local and global features of the image to obtain multi-scale features, construct the dependency relationship between global space and channels, and then train a meta-model containing specific distortion knowledge; A spatial pyramid pooling layer is added after the meta-model containing specific distortion knowledge to receive images of any size; The multi-scale features are globally pooled to obtain a feature set, which is then fed into a node construction module and an edge construction module to obtain a graph representation of a specific distortion type; The nodes and edges of the graph representation of the specific distortion type are input into a distortion type discrimination module to obtain a triple loss, and the contrast relationship between different distortions is learned; The mean square error loss is obtained through the distortion level discrimination module, and the overall model is optimized by weighting the triple loss to obtain prior knowledge for constructing the distortion graph representation; In the fine-tuning stage, the distortion features of the distorted image are extracted by the pre-trained feature extraction network; The node construction module and the edge construction module with prior knowledge are used to quickly obtain a good representation of the distortion features; The good representation of the distortion features is mapped to the corresponding quality score through a fully connected layer.

2. The method of claim 1, wherein the method is based on graph convolution and multi-scale features. When the Resnet backbone network is trained by a double-layer gradient optimization to adapt to unknown distortions, an adaptive multi-scale feature fusion module is introduced to divide the convolutional blocks into multiple parts, calibrate the feature changes of one part of the convolutional blocks through the transformation of another part of the convolutional blocks, and adaptively construct the dependency relationship between global space and channels.

3. The method of claim 2, wherein the method is based on graph convolution and multi-scale features. The adaptive fusion of local and global features of the image obtains multi-scale features and constructs the dependency relationship between global space and channels, which comprises: For a given set of feature layers with shape C, H, W, C, H, and W represent the number of channels, height, and width, respectively, the set is divided into three convolutional blocks M1, M2, and M3; The input feature X is divided into two parts, X local global respectively represent local features and global features;​​ Using M1, M2 on X global Perform calibration operation, given input X global Use average pooling of size R x R, stride R to expand receptive field: P1 = AVGPool R (X global ) The feature set is fed into a node construction module and an edge construction module to obtain a graph representation of a specific distortion type, which comprises: X' global = UP(P1 x M1) In the formula, UP(·) is a bilinear interpolation operator, and a calibration operation is added after feature transformation; X global Channel enhancement is performed: Y′=(X global ×M2)·σ(X global +X′ global ) The node construction module is composed of multiple fully connected layers, and the specific expression of inputting the feature set into the node construction module is as follows: Using X' global As residual factors, global distortion information is captured, and the captured global distortion information is fused with the original feature X local The output vector Y is obtained by fusion: In the formulae, denotes a characteristic-wise addition.

4. The method of claim 1, wherein the method is based on graph convolution and multi-scale features. The feature set is fed into a node construction module and an edge construction module to obtain a graph representation of a specific distortion type, which comprises: After initializing the edge features, the edge construction module composed of graph convolutional networks is input to further optimize the internal structure of the graph representation, and the specific process is as follows: where f k,i represents the feature vector of each image, represents the node feature dimension, N represents the number of distorted images, F N (f k,i ; θ) represents the node construction network, θ represents the network parameters of F N , v k,i represents the node generated by the i-th sample in the distortion type k, V k represents the node feature set represented by the distorted image.

5. The method of claim 1, wherein the method is based on graph convolution and multi-scale features. The edge feature set of the distortion graph representation can be represented as: Initialize edge features by the dot product between nodes The value of the edge feature is denoted as denotes the dimension of node features, i, j denotes the relationship between the ith sample and the jth sample. After representing the distortion image as a graph structure, the global information from the node feature set and the inter-sample relationship from the edge feature set are aggregated by the distortion type discrimination module to obtain the specific expression of the triple loss function as follows: initialized edge features and the corresponding adjacency matrix As the edge construction module input, the value of the edge feature optimized by the l-layer graph convolution network can be calculated as: In the formula, σ represents a ReLu operation, represents a three-dimensional adjacency matrix, I represents a unit matrix, represents the output of the lth layer graph convolutional network, is the weight matrix of the lth layer of the graph convolutional network; ​ in, represents the edge feature dimension, e k,i,j Indicates the relationship between the i-th sample and the j-th sample, E k represents the edge feature set represented by the distortion graph, and N represents the number of distorted images.

6. The method of claim 5, wherein the method is based on graph convolution and multi-scale features. ​ Among them, l represents the number of layers of the distortion type discrimination module composed of the graph convolutional network, Represents node V k The lth and l+1th layers, σ represents the ReLu operation, T represents the distortion type discrimination module, Indicates E k Perform average pooling transformation to obtain node V k The two-dimensional adjacency matrix of is the weight matrix of the lth layer of the graph convolutional network, y code represents the output vector of the distortion type discrimination module, d represents the Euclidean distance, Represents and y code Samples belonging to the same type are positive samples, and Represents and y code Different types of samples represent negative samples, and margin represents the threshold that separates positive and negative samples when comparing. represents the triplet loss function.

7. The method of claim 1, wherein the method is based on graph convolution and multi-scale features. The edge feature E k By averaging the input after pooling, the information from the current node and the remaining adjacent nodes is aggregated, and the distortion level discrimination module is trained using the mean square error loss function. The specific expression is: where y i and y′ i denote the predicted and true distortion degree respectively, i denotes the i-th sample, denotes the mean square error function, denotes the triplet loss function, the two loss functions and are weighted by the hyper-parameter λ to obtain the loss function of the overall model 8. The method of claim 1, wherein the method is based on graph convolution and multi-scale features. In the fine-tuning stage, the outputs of the node construction module and the edge construction module with rich prior knowledge are spliced and then input into a fully connected layer to predict the final quality score. The mean square error function of the trained model is specifically: where N f represents the mini-batch size of the training, y i and y′ i represent the predicted and true distortion degree, respectively, and i represents the i-th sample.

9. The method of claim 1, wherein the method is based on graph convolution and multi-scale features. The backbone network is a Resnet50 feature extraction network.

10. A storage medium having stored thereon a computer program, characterized in that The computer program, when executed by a processor, implements the steps of the method for no-reference image quality evaluation based on graph convolution and multi-scale features according to any one of claims 1-9. The computer program, when executed by a processor, implements the steps of the method for no-reference image quality evaluation based on graph convolution and multi-scale features according to any one of claims 1-9.

Images