No-reference image quality evaluation method and system based on weak correlation knowledge distillation

Abstract

The invention discloses a non-reference image quality evaluation method and system based on weak correlation knowledge distillation, and the method comprises the steps: inputting training samples into a quality evaluation model and a semantic segmentation model for feature extraction, and obtaining multi-scale quality features and multi-scale semantic features; correlation constraint is carried out on the multi-scale quality features and the multi-scale semantic features, and correlation constraint loss is calculated; fusing the multi-scale quality features and the multi-scale semantic features, and performing quality regression on the fused features to obtain a prediction score; calculating an overall loss function according to the prediction score and the correlation constraint loss, and updating the weight of the quality evaluation model according to the loss function to obtain a target quality evaluation model; and inputting a to-be-evaluated image into the target quality evaluation model for calculation, and outputting a quality score of the to-be-evaluated image. According to the method, weak correlation between semantic segmentation and a BIQA task is utilized, the representation learning ability of quality perception features is enhanced, and the image quality evaluation accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular, to a no-reference image quality assessment method, system, terminal and computer-readable storage medium based on weakly correlated knowledge distillation. Background Art

[0002] High-quality images are an indispensable foundation for image processing and computer vision technologies, and play an important role in promoting the accuracy, efficiency of algorithms and the expansion of application scenarios. During the processes of acquisition, processing, transmission and storage, due to imperfections in imaging systems, processing methods, transmission media and recording devices, etc., images will inevitably introduce interference and noise, thereby causing different types and degrees of distortion to their quality, resulting in a decline in image quality and affecting the visual experience of end users and the reliability and accuracy of subsequent computer vision tasks. Image Quality Assessment (IQA) refers to quantifying the visual distortion degree of an image by analyzing relevant characteristics of the image, so as to evaluate the quality of the image. It can be used to dynamically detect and adjust the image quality status, and can also be used as a basis for algorithm or parameter optimization in an image processing system. It has wide practicability in the fields of image restoration, image compression, video encoding and decoding, etc. However, there are still problems such as low efficiency and limited data samples in IQA.

[0003] Current Blind Image Quality Assessment (BIQA) methods based on knowledge distillation improve the accuracy of BIQA models by extracting knowledge from tasks closely related to BIQA and transferring it to the target BIQA model. However, visual tasks with relatively weak correlation with IQA are often overlooked. For example, semantic segmentation models are sensitive to semantic content but not to image quality; while IQA models are exactly the opposite. In addition, the same distortion may lead to different degrees of quality degradation due to different semantic contents. Despite the significant progress made in semantic segmentation, its application in BIQA is still less.

[0004] Therefore, the prior art still needs to be improved and developed. Summary of the Invention

[0005] The main purpose of the present invention is to provide a no-reference image quality assessment method, system, terminal and computer-readable storage medium based on weakly correlated knowledge distillation, aiming to solve the problem in the prior art that in blind image quality assessment based on knowledge distillation, visual tasks with relatively weak correlation with IQA are often overlooked, resulting in poor quality perception effect and low accuracy of image quality assessment.

[0006] To achieve the above object, the present invention provides a no-reference image quality evaluation method based on weakly correlated knowledge distillation. The no-reference image quality evaluation method based on weakly correlated knowledge distillation includes the following steps:

[0007] Input the distorted images in the training samples into a quality evaluation model and a semantic segmentation model respectively for feature extraction to obtain multi-scale quality features and multi-scale semantic features;

[0008] Perform correlation constraints on the multi-scale quality features and the multi-scale semantic features, and calculate the correlation constraint loss;

[0009] Fuse the multi-scale quality features and the multi-scale semantic features to obtain fused features, and perform quality regression on the fused features to obtain a predicted score;

[0010] Calculate an overall loss function according to the predicted score and the correlation constraint loss, and update the weights of the quality evaluation model according to the loss function to obtain a target quality evaluation model;

[0011] Input the image to be evaluated into the target quality evaluation model for calculation, and output the quality score of the image to be evaluated.

[0012] Optionally, in the no-reference image quality evaluation method based on weakly correlated knowledge distillation, the backbone network types of the quality evaluation model and the semantic segmentation model are the same to maintain the alignment of the dimensions of the feature representations.

[0013] Optionally, in the no-reference image quality evaluation method based on weakly correlated knowledge distillation, the performing correlation constraints on the multi-scale quality features and the multi-scale semantic features, and calculating the correlation constraint loss specifically includes:

[0014] Calculate the first covariance matrix of the multi-scale quality features and the second covariance matrix of the multi-scale semantic features respectively, and calculate the difference matrix between the first covariance matrix and the second covariance matrix of the multi-scale semantic features;

[0015] Deploy a clustering algorithm to the difference matrix, cluster and group the elements on the difference matrix to obtain a grouping result, and generate a first mask matrix and a second mask matrix according to the grouping result;

[0016] Calculate the whitening loss according to the first mask matrix and the first covariance matrix, calculate the normalization loss according to the second mask matrix, and calculate the correlation constraint loss according to the whitening loss and the normalization loss.

[0017] Optionally, for the no-reference image quality assessment method based on weakly correlated knowledge distillation, calculating the first covariance matrix of the multi-scale quality features and the second covariance matrix of the multi-scale semantic features, and calculating the difference matrix between the first covariance matrix and the second covariance matrix of the multi-scale semantic features specifically includes:

[0018] Flatten the multi-scale quality features and the multi-scale semantic features to obtain the flattened quality features and the flattened semantic features;

[0019] Calculate the first covariance matrix of the flattened quality features and the second covariance matrix of the flattened semantic features:

[0020]

[0021] Wherein, is the first covariance matrix, is the second covariance matrix, h and w are the height and width of the corresponding features respectively, F q ' is the flattened quality features, F s ' is the flattened semantic features, is the transpose operation;

[0022] According to the upper triangular matrices of the first covariance matrix and the second covariance matrix, calculate the difference matrix between the first covariance matrix and the second covariance matrix of the multi-scale semantic features:

[0023]

[0024] Wherein, D is the difference matrix, || + represents taking the upper triangular part of the difference matrix.

[0025] Optionally, for the no-reference image quality assessment method based on weakly correlated knowledge distillation, deploying a clustering algorithm to the difference matrix, clustering and grouping the elements on the difference matrix to obtain a grouping result, and generating a first mask matrix and a second mask matrix according to the grouping result specifically includes:

[0026] Deploy the k-means clustering algorithm to the difference matrix, cluster the elements on the difference matrix into k clusters, and divide the k clusters into a low difference value group and a high difference value group according to a preset threshold;

[0027] Generate a first mask matrix and a second mask matrix according to the low difference value group and the high difference value group:

[0028]

[0029] Among them, M l is the first mask matrix, and M h is the second mask matrix. D x,y represents the element at the x - coordinate and y - coordinate in the difference matrix. G low is the low - difference value group, and G high is the high - difference value group.

[0030] Optionally, in the no - reference image quality assessment method based on weakly - correlated knowledge distillation, the whitening loss is calculated according to the first mask matrix and the first covariance matrix, the normalization loss is calculated according to the second mask matrix, and the correlation constraint loss is calculated according to the whitening loss and the normalization loss. Specifically, it includes:

[0031] Multiply the first mask matrix by the first covariance matrix to calculate the whitening loss to suppress the covariance term of low - difference values:

[0032]

[0033] Calculate the normalization loss according to the second mask matrix to expand the feature distance:

[0034] L h = E[|D⊙M h - M h |1];

[0035] Add the whitening loss and the normalization loss to obtain the correlation constraint loss:

[0036] L cc = L l + L h ;

[0037] Among them, L l is the whitening loss, L h is the normalization loss, L cc is the correlation constraint loss, ⊙ is the Hadamard product, ||1 means to accumulate the elements after taking the absolute value, and E is the expectation.

[0038] Optionally, in the no - reference image quality assessment method based on weakly - correlated knowledge distillation, the calculation of the overall loss function according to the prediction score and the correlation constraint loss specifically includes:

[0039] Obtain the quality assessment task loss according to the prediction score and the average subjective score corresponding to the prediction score;

[0040] Calculate the overall loss function according to the quality assessment task loss and the correlation constraint loss:

[0041]

[0042] Among them, L total is the overall loss function, N is the batch size, M is all feature layers adding the correlation constraint loss, i is the ordinal number of the sample, j corresponds to the ordinal number of the feature layer, and λ is the weight of the correlation constraint loss. is the loss of the quality evaluation task for the i-th sample. The correlation constraint loss of the j-th feature layer of the i-th sample.

[0043] In addition, to achieve the above object, the present invention also provides a no-reference image quality evaluation system based on weak correlation knowledge distillation. Among them, the no-reference image quality evaluation system based on weak correlation knowledge distillation includes:

[0044] A multi-scale feature extraction module for respectively inputting the distorted images in the training samples into a quality evaluation model and a semantic segmentation model for feature extraction to obtain multi-scale quality features and multi-scale semantic features;

[0045] A multi-scale correlation constraint module for performing correlation constraints on the multi-scale quality features and the multi-scale semantic features and calculating the correlation constraint loss;

[0046] A feature fusion regression module for fusing the multi-scale quality features and the multi-scale semantic features to obtain fusion features and performing quality regression on the fusion features to obtain a prediction score;

[0047] A quality evaluation model update module for calculating an overall loss function according to the prediction score and the correlation constraint loss and updating the weights of the quality evaluation model according to the loss function to obtain a target quality evaluation model;

[0048] A quality score prediction module for inputting the image to be evaluated into the target quality evaluation model for calculation and outputting the quality score of the image to be evaluated.

[0049] In addition, to achieve the above object, the present invention also provides a terminal. Among them, the terminal includes: a memory, a processor, and a no-reference image quality evaluation program based on weak correlation knowledge distillation stored on the memory and executable on the processor. When the no-reference image quality evaluation program based on weak correlation knowledge distillation is executed by the processor, the steps of the no-reference image quality evaluation method based on weak correlation knowledge distillation as described above are implemented.

[0050] In addition, to achieve the above object, the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a no-reference image quality evaluation program based on weak correlation knowledge distillation, and when the no-reference image quality evaluation program based on weak correlation knowledge distillation is executed by a processor, the steps of the no-reference image quality evaluation method based on weak correlation knowledge distillation as described above are implemented.

[0051] In the present invention, training samples are respectively input into a quality evaluation model and a semantic segmentation model for feature extraction to obtain multi-scale quality features and multi-scale semantic features; correlation constraints are imposed on the multi-scale quality features and multi-scale semantic features, and a correlation constraint loss is calculated; the multi-scale quality features and multi-scale semantic features are fused, and the fused features are subjected to quality regression to obtain a predicted score; an overall loss function is calculated according to the predicted score and the correlation constraint loss, and the weights of the quality evaluation model are updated according to the loss function to obtain a target quality evaluation model; the image to be evaluated is input into the target quality evaluation model for calculation, and the quality score of the image to be evaluated is output. The present invention utilizes the weak correlation between semantic segmentation and the BIQA task to enhance the representation learning ability of quality-aware features and improve the accuracy of image quality evaluation. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 is a flowchart of a preferred embodiment of the no-reference image quality evaluation method based on weak correlation knowledge distillation of the present invention;

[0053] Figure 2 is an overall architecture diagram of the no-reference image quality evaluation method based on weak correlation knowledge distillation of the present invention;

[0054] Figure 3 is a schematic diagram of the correlation constraint of the no-reference image quality evaluation method based on weak correlation knowledge distillation of the present invention;

[0055] Figure 4 is a structural diagram of a preferred embodiment of the no-reference image quality evaluation system based on weak correlation knowledge distillation of the present invention;

[0056] Figure 5 is a structural diagram of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0057] The present application provides a no-reference image quality evaluation method, system and terminal based on weak correlation knowledge distillation. To make the purpose, technical solution and effect of the present application clearer and more definite, the following further describes the present application in detail with reference to the accompanying drawings and by way of examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0058] Those skilled in the art can understand that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as the general understanding of those of ordinary skill in the field to which this application belongs. It should also be understood that terms such as those defined in a general dictionary should be understood to have a meaning consistent with the meaning in the context of the prior art, and will not be interpreted in an idealized or overly formal sense unless specifically defined as here.

[0059] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one such feature. In addition, the technical solutions between various embodiments can be combined with each other, but it must be based on the ability of those of ordinary skill in the art to implement. When the combination of technical solutions results in contradictions or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0060] The method for no-reference image quality assessment based on weakly correlated knowledge distillation described in the preferred embodiment of the present invention, as Figure 1 and Figure 2 shown, the method for no-reference image quality assessment based on weakly correlated knowledge distillation includes the following steps:

[0061] Step S10: Input the distorted images in the training samples into a quality assessment model and a semantic segmentation model respectively for feature extraction, to obtain multi-scale quality features and multi-scale semantic features.

[0062] The present invention proposes to adopt a weakly correlated knowledge distillation method to fully exploit the weak correlation between the semantic segmentation and quality assessment tasks, and transfer the knowledge learned by the semantic segmentation model to the quality assessment model. First, a pre-trained semantic segmentation model is introduced, and then a feature correlation constraint is added between the features of different scales of the quality assessment model and the segmentation model, so as to widen the distance between the features of the two models, enabling the quality assessment model to learn more quality-related and semantic-irrelevant information. Subsequently, the semantic features and quality features are fused, and the quality is completed with the fused features.

[0063] Specifically, the training samples are images of a quality assessment data set. There are two types of data sets. One is a synthetic distortion data set, and the other is a real distortion data set. Synthetic distorted images generally refer to distorted images obtained by artificially adding various types of distortions to a non-distorted reference image; real distorted images are some images taken in real life with jitter, blur or overexposure, and generally have relatively complex distortion characteristics.

[0064] In this embodiment, the label corresponding to the training sample is MOS (Mean Opinion Score). Observers are required to make a quality judgment on the image to be evaluated according to the pre - defined evaluation criteria or their own subjective experience in a specific experimental environment, and give a quality score. Finally, the quality scores given by all evaluators are weighted and averaged to obtain the mean opinion score of the image.

[0065] For a distorted image X ∈ R C×H×W , where H, W, and C are the height, width, and number of channels of the distorted image respectively, the purpose of the image quality evaluation model f q is to predict the quality score of X. The present invention introduces a semantic segmentation model f s and ensures that it has the same type of backbone network as the quality evaluation model, so as to keep the dimensional alignment of feature representations.

[0066] Furthermore, multi - scale features are extracted from the image quality evaluation model f q and the semantic segmentation model f s respectively, and are characterized as where h, w, and c represent the height, width, and number of channels of the feature image respectively, R is the spatial dimension, i represents the serial number of the feature output layer, F q is the multi - scale quality feature, and F s is the multi - scale semantic feature. It can be understood that the image quality evaluation model and the semantic segmentation model are existing models, and are optimized by adding a weakly - related knowledge distillation method to the existing models.

[0067] Step S20: Perform correlation constraints on the multi - scale quality feature and the multi - scale semantic feature, and calculate the correlation constraint loss.

[0068] The performing correlation constraints on the multi - scale quality feature and the multi - scale semantic feature, and calculating the correlation constraint loss specifically includes:

[0069] Calculate the first covariance matrix of the multi - scale quality feature and the second covariance matrix of the multi - scale semantic feature respectively, and calculate the difference matrix between the first covariance matrix and the second covariance matrix of the multi - scale semantic feature;

[0070] Deploy a clustering algorithm to the difference matrix, cluster and group the elements on the difference matrix to obtain a grouping result, and generate a first mask matrix and a second mask matrix according to the grouping result;

[0071] Calculate the whitening loss based on the first mask matrix and the first covariance matrix, calculate the normalization loss based on the second mask matrix, and calculate the correlation constraint loss based on the whitening loss and the normalization loss.

[0072] Further, the method for separately calculating the first covariance matrix of the multi-scale quality features and the second covariance matrix of the multi-scale semantic features, and calculating the difference matrix between the first covariance matrix and the second covariance matrix of the multi-scale semantic features specifically includes:

[0073] Flatten the multi-scale quality features and the multi-scale semantic features to obtain the flattened quality features and the flattened semantic features;

[0074] Calculate the first covariance matrix of the flattened quality features and the second covariance matrix of the flattened semantic features:

[0075]

[0076] Wherein, is the first covariance matrix, is the second covariance matrix, h and w are the height and width of the corresponding features respectively, and F q ' is the flattened quality feature, and F s ' is the flattened semantic feature, and T represents the transpose operation;

[0077] Calculate the difference matrix between the first covariance matrix and the second covariance matrix of the multi-scale semantic features according to the upper triangular matrix of the first covariance matrix and the second covariance matrix:

[0078]

[0079] Wherein, D is the difference matrix, and || + represents taking the upper triangular part of the difference matrix.

[0080] As Figure 3 shown in (a) of, it can be understood that after transposing and multiplying the flattened features, the result is normalized by dividing by hw. The elements of the covariance matrix represent the covariance between the corresponding channel features, which can help analyze the correlation between the image channel features. It can be found that the diagonal elements are 1, and the covariance matrix is symmetric up and down. Therefore, the upper triangular matrix of the covariance matrix is taken to obtain the difference matrix, which represents the differences in the image information learned by the segmentation model and the quality evaluation model. The high-difference terms carry task-specific information.

[0081] Further, deploying the clustering algorithm to the difference matrix, clustering and grouping the elements on the difference matrix to obtain a grouping result, and generating a first mask matrix and a second mask matrix according to the grouping result specifically includes:

[0082] Deploy the k-means clustering algorithm to the difference matrix, cluster the elements on the difference matrix into k clusters, and divide the k clusters into a low-difference-value group and a high-difference-value group according to a preset threshold;

[0083] Generate a first mask matrix and a second mask matrix according to the low-difference-value group and the high-difference-value group:

[0084]

[0085] Among them, M l is the first mask matrix, M h is the second mask matrix, D x,y represents the element with abscissa x and ordinate y in the difference matrix, G low is the low-difference-value group, G high is the high-difference-value group.

[0086] In this embodiment, in order to identify these elements and maximize task uniqueness, the k-means clustering algorithm is deployed to the difference matrix to distill information related to quality but irrelevant to semantics, and the elements on D are clustered into k clusters C = {c1, c2,..., c k}, and a threshold is set to divide these clusters into two groups: G low = {c1,..., c n} representing low difference values and G high = {c n+1 ,..., c k} representing high difference values. Based on the grouping result, two mask matrices M l , M h ∈R C×C are generated. In the formula for generating the mask matrix above, the elements in the corresponding clustering group are set to 1, and the rest are 0.

[0087] Further, calculating the whitening loss according to the first mask matrix and the first covariance matrix, calculating the normalization loss according to the second mask matrix, and calculating the correlation constraint loss according to the whitening loss and the normalization loss specifically includes:

[0088] Multiply the first mask matrix by the first covariance matrix to calculate the whitening loss to suppress the covariance terms of low difference values:

[0089]

[0090] Calculate the normalized loss according to the second mask matrix to expand the feature distance:

[0091] L h = E[|D⊙M h - M h |1];

[0092] Add the whitening loss and the normalized loss to obtain the correlation constraint loss:

[0093] L cc = L l + L h ;

[0094] where L l is the whitening loss, L h is the normalized loss, L cc is the correlation constraint loss, ⊙ is the Hadamard product, ||1 means to accumulate the elements after taking the absolute value, and E is the expectation.

[0095] As shown in (b) of Figure 3 , in this embodiment, multiply the obtained first mask matrix M l by the first covariance matrix corresponding to the quality feature to calculate its whitening loss L l to suppress the covariance terms with low difference values. On the other hand, as shown in (c) of Figure 3 , select the high-difference terms according to the second mask matrix M h , normalize the entire difference matrix, and set the normalized loss L h to make each high-difference term close to 1, thereby expanding the feature distance between the multi-scale quality feature and the multi-scale semantic feature, and the whitening loss and the normalized loss together constitute the correlation constraint loss L cc .

[0096] Step S30: Fuse the multi-scale quality feature and the multi-scale semantic feature to obtain a fused feature, and perform quality regression on the fused feature to obtain a prediction score.

[0097] Specifically, the weak correlation between semantic segmentation and quality evaluation also means that the same distortion will cause different degrees of quality degradation in different semantic contents. Therefore, the rich scene information learned by the segmentation model is also helpful for the quality evaluation task. Therefore, the present invention fuses the features of the two models to further enhance the ability of the BIQA model to perceive image quality under different semantic contents. And perform quality regression on the fused feature to obtain a prediction score. Among them, the formula for fusing the multi-scale quality feature and the multi-scale semantic feature to obtain a fused feature is:

[0098] S = Conv(Concat(F q , F s ))), where S is the fused feature, Concat() is the fusion operation, and Conv() is the convolution operation.

[0099] Step S40: Calculate the overall loss function according to the predicted score and the correlation constraint loss, and update the weights of the quality evaluation model according to the loss function to obtain the target quality evaluation model.

[0100] The calculation of the overall loss function according to the predicted score and the correlation constraint loss specifically includes:

[0101] Obtain the quality evaluation task loss according to the predicted score and the average subjective score corresponding to the predicted score;

[0102] Calculate the overall loss function according to the quality evaluation task loss and the correlation constraint loss:

[0103]

[0104] where L total is the overall loss function, N is the batch size, M is all the feature layers to which the correlation constraint loss is added, i is the ordinal number of the sample, j corresponds to the ordinal number of the feature layer, λ is the weight of the correlation constraint loss, is the quality evaluation task loss of the i-th sample, the correlation constraint loss of the j-th feature layer of the i-th sample.

[0105] Furthermore, update the weights of the quality evaluation model according to the loss function to obtain the target quality evaluation model. In this embodiment, it is trained according to the set number of epochs (an epoch in a neural network represents the training process of passing through all the samples in the training dataset once), and a test is performed after each epoch of training is completed. The model with the best SRCC (Spearman rank-order correlation coefficient) and PLCC (Pearson correlation coefficient) in one round is taken as the optimal model, that is, the target quality evaluation model.

[0106] Step S50: Input the image to be evaluated into the target quality evaluation model for calculation, and output the quality score of the image to be evaluated.

[0107] Specifically, the present invention is a general optimization method that can be added to most existing BIQA models to optimize the BIQA model, that is, to obtain a target quality assessment model. The target quality assessment model can receive an image to be evaluated, extract features of the image to be evaluated and perform regression prediction, and output the quality score of the image to be evaluated.

[0108] It can be understood that the above no-reference image quality assessment method based on weakly correlated knowledge distillation is deployed on multiple quality assessment models, and the accuracy has been improved to a certain extent on multiple data sets. Select several basic feature extraction backbone networks: ResNet50, ViT-B-16 and VGG16, and multiple existing image quality assessment models: HyperIQA, TReS and LIQE as the image quality assessment models to be optimized, and introduce multiple pre-trained segmentation models such as DeepLabV3, DeepLabV1, DinoV2 to assist the IQA model to complete weakly correlated knowledge distillation. Keep the original training parameters of each model unchanged and conduct training, and use the obtained results as a benchmark, and add the weakly correlated knowledge distillation method for comparison to verify the effect of this method.

[0109] Select seven existing quality assessment data sets, namely LIVE, CSIQ, TID2013, KADID-10K, CLIVE, KonIQ-10K, and BID, and conduct training respectively. The results are shown in Table 1 below. Table 1 is a schematic diagram of the experimental results of the no-reference image quality assessment method based on weakly correlated knowledge distillation of the present invention on multiple data sets; using the Spearman rank correlation coefficient and the Pearson linear correlation coefficient as evaluation indicators to test the quality prediction effect of the model. Judging from the results, the weakly correlated knowledge distillation method has improved the prediction accuracy of the model to a certain extent. For example, the comprehensive SROCC of TReS has increased by 2.7%, and the PLCC has increased by 2.6%; the comprehensive SROCC of HyperIQA has increased by 2.1%, and the PLCC has increased by 1.8%; the comprehensive SROCC of LIQE has increased by 1%, and the PLCC has increased by 1.1%.

[0110] Table 1. Schematic diagram of experimental results

[0111]

[0112] It can be seen that the present invention first introduces a pre-trained semantic segmentation model that uses the same backbone network as the BIQA model, and then designs a correlation constraint to maximize the distance between the features extracted by the semantic segmentation model and the BIQA model, so that the BIQA model can learn information that is irrelevant to semantics but relevant to image quality. In addition, since the quality perception effects of the same distortion are different under different semantic contents, the present invention further fuses the features of the semantic segmentation model and the BIQA model to better represent the visual perception quality of the image. Experimental results show that the method of the present invention has significant advantages in testing on multiple state-of-the-art BIQA models and datasets. The present invention specifically includes the following beneficial effects:

[0113] (1) The present invention proposes a weakly correlated knowledge distillation framework, which utilizes the weak correlation between semantic segmentation and BIQA tasks to enhance the representation learning ability of quality perception features.

[0114] (2) The present invention designs a special multi-scale correlation constraint module for extracting features that are irrelevant to semantics but relevant to image quality, effectively differentiating the feature representations of semantic segmentation and BIQA tasks.

[0115] (3) The present invention designs a quality-semantics fusion module to fuse the scene content information and quality-related features extracted from the semantic segmentation model, improving the robustness and generalization ability of the BIQA model in various scenarios.

[0116] (4) The present invention deploys this method on multiple existing BIQA models and conducts extensive experiments on multiple quality evaluation datasets for synthetic and real distortions respectively. The results show that this method has significant superiority.

[0117] Furthermore, as Figure 4 shown, based on the above-mentioned no-reference image quality assessment method based on weakly correlated knowledge distillation, the present invention also correspondingly provides a no-reference image quality assessment system based on weakly correlated knowledge distillation, wherein the no-reference image quality assessment system based on weakly correlated knowledge distillation includes:

[0118] A multi-scale feature extraction module 51, configured to input the distorted images in the training samples into a quality assessment model and a semantic segmentation model respectively for feature extraction, to obtain multi-scale quality features and multi-scale semantic features;

[0119] A multi-scale correlation constraint module 52, configured to perform correlation constraint on the multi-scale quality features and the multi-scale semantic features, and calculate a correlation constraint loss;

[0120] The feature fusion regression module 53 is used to fuse the multi-scale quality features and the multi-scale semantic features to obtain fused features, and perform quality regression on the fused features to obtain a prediction score;

[0121] The quality evaluation model update module 54 is used to calculate an overall loss function according to the prediction score and the correlation constraint loss, and update the weights of the quality evaluation model according to the loss function to obtain a target quality evaluation model;

[0122] The quality score prediction module 55 is used to input the image to be evaluated into the target quality evaluation model for calculation, and output the quality score of the image to be evaluated.

[0123] Further, as Figure 5 shown, based on the above no-reference image quality evaluation method and system based on weak correlation knowledge distillation, the present invention also correspondingly provides a terminal, which includes a processor 10, a memory 20, and a display 30. Figure 5 Only some components of the terminal are shown, but it should be understood that it is not required to implement all the shown components, and more or fewer components can be alternatively implemented.

[0124] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as the hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk equipped on the terminal, a SmartMedia Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. Further, the memory 20 may also include both the internal storage unit and the external storage device of the terminal. The memory 20 is used to store application software installed on the terminal and various types of data, such as program codes installed on the terminal, etc. The memory 20 may also be used to temporarily store data that has been output or will be output. In one embodiment, a no-reference image quality evaluation program 40 based on weak correlation knowledge distillation is stored on the memory 20, and the no-reference image quality evaluation program 40 based on weak correlation knowledge distillation can be executed by the processor 10, so as to implement the no-reference image quality evaluation method in the present application.

[0125] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chips, and is used to run program codes stored in the memory 20 or process data, such as executing the no-reference image quality evaluation method based on weak correlation knowledge distillation, etc.

[0126] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 30 is used to display information on the terminal and to display a visual user interface. Components of the terminal communicate with each other via a system bus.

[0127] In one embodiment, when the processor 10 executes the no-reference image quality evaluation program 40 based on weak correlation knowledge distillation in the memory 20, the following steps are implemented:

[0128] The distorted images in the training samples are respectively input into a quality evaluation model and a semantic segmentation model for feature extraction to obtain multi-scale quality features and multi-scale semantic features;

[0129] Perform correlation constraints on the multi-scale quality features and the multi-scale semantic features, and calculate a correlation constraint loss;

[0130] Fuse the multi-scale quality features and the multi-scale semantic features to obtain fused features, and perform quality regression on the fused features to obtain a predicted score;

[0131] Calculate an overall loss function according to the predicted score and the correlation constraint loss, and update the weights of the quality evaluation model according to the loss function to obtain a target quality evaluation model;

[0132] Input the image to be evaluated into the target quality evaluation model for calculation, and output the quality score of the image to be evaluated.

[0133] Wherein, the backbone network types of the quality evaluation model and the semantic segmentation model are the same to keep the dimension alignment of the feature representations.

[0134] Wherein, performing correlation constraints on the multi-scale quality features and the multi-scale semantic features, and calculating a correlation constraint loss specifically includes:

[0135] Calculate a first covariance matrix of the multi-scale quality features and a second covariance matrix of the multi-scale semantic features respectively, and calculate a difference matrix between the first covariance matrix and the second covariance matrix of the multi-scale semantic features;

[0136] Deploy a clustering algorithm to the difference matrix, cluster and group the elements on the difference matrix to obtain a grouping result, and generate a first mask matrix and a second mask matrix according to the grouping result;

[0137] Calculate the whitening loss based on the first mask matrix and the first covariance matrix, calculate the normalization loss based on the second mask matrix, and calculate the correlation constraint loss based on the whitening loss and the normalization loss.

[0138] Among them, the first covariance matrix of the multi-scale quality features and the second covariance matrix of the multi-scale semantic features are respectively calculated, and the difference matrix between the first covariance matrix and the second covariance matrix of the multi-scale semantic features is calculated. Specifically, it includes:

[0139] Flatten the multi-scale quality features and the multi-scale semantic features to obtain the flattened quality features and the flattened semantic features;

[0140] Calculate the first covariance matrix of the flattened quality features and the second covariance matrix of the flattened semantic features:

[0141]

[0142] Among them, is the first covariance matrix, is the second covariance matrix, h and w are the height and width of the corresponding features respectively, and F q ' is the flattened quality features, and F s ' is the flattened semantic features, is the transpose operation;

[0143] Calculate the difference matrix between the first covariance matrix and the second covariance matrix of the multi-scale semantic features according to the upper triangular matrices of the first covariance matrix and the second covariance matrix:

[0144]

[0145] Among them, D is the difference matrix, and || + represents taking the upper triangular part of the difference matrix.

[0146] Among them, deploying the clustering algorithm to the difference matrix, clustering and grouping the elements on the difference matrix to obtain the grouping result, and generating the first mask matrix and the second mask matrix according to the grouping result. Specifically, it includes:

[0147] Deploy the k-means clustering algorithm to the difference matrix, cluster the elements on the difference matrix into k clusters, and divide the k clusters into a low difference value group and a high difference value group according to a preset threshold;

[0148] Generate the first mask matrix and the second mask matrix according to the low difference value group and the high difference value group:

[0149]

[0150]

[0151] Among them, M l is the first mask matrix, M h is the second mask matrix, D x,y represents the element at the abscissa x and ordinate y in the difference matrix, G low is the low-difference value group, G high is the high-difference value group.

[0152] Among them, calculating the whitening loss according to the first mask matrix and the first covariance matrix, calculating the normalization loss according to the second mask matrix, and calculating the correlation constraint loss according to the whitening loss and the normalization loss specifically include:

[0153] Multiplying the first mask matrix by the first covariance matrix to calculate the whitening loss to suppress the covariance term of the low-difference value:

[0154]

[0155] Calculating the normalization loss according to the second mask matrix to expand the feature distance:

[0156] L h = E[|D⊙M h -M h |1];

[0157] Adding the whitening loss and the normalization loss to obtain the correlation constraint loss:

[0158] L cc = L l +L h ;

[0159] Among them, L l is the whitening loss, L h is the normalization loss, L cc is the correlation constraint loss, ⊙ is the Hadamard product, ||1 means taking the absolute value of the elements and then accumulating, and E is the expectation.

[0160] Among them, calculating the overall loss function according to the prediction score and the correlation constraint loss specifically includes:

[0161] Obtaining the quality evaluation task loss according to the prediction score and the average subjective score corresponding to the prediction score;

[0162] Calculate the overall loss function according to the quality evaluation task loss and the relevance constraint loss:

[0163]

[0164] where L total is the overall loss function, N is the batch size, M is all the feature layers to which the relevance constraint loss is added, i is the ordinal number of the sample, j is the ordinal number of the corresponding feature layer, λ is the weight of the relevance constraint loss, is the quality evaluation task loss of the i-th sample, is the relevance constraint loss of the j-th feature layer of the i-th sample.

[0165] The present invention also provides a computer-readable storage medium. The computer-readable storage medium stores a no-reference image quality evaluation program based on weak correlation knowledge distillation. When the no-reference image quality evaluation program based on weak correlation knowledge distillation is executed by a processor, the steps of the no-reference image quality evaluation method based on weak correlation knowledge distillation as described above are implemented.

[0166] In summary, the present invention proposes a no-reference image quality evaluation method and system based on weak correlation knowledge distillation. The method includes: respectively inputting the distorted images in the training samples into a quality evaluation model and a semantic segmentation model for feature extraction to obtain multi-scale quality features and multi-scale semantic features; performing relevance constraints on the multi-scale quality features and the multi-scale semantic features, and calculating the relevance constraint loss; fusing the multi-scale quality features and the multi-scale semantic features to obtain fused features, and performing quality regression on the fused features to obtain a prediction score; calculating an overall loss function according to the prediction score and the relevance constraint loss, and updating the weights of the quality evaluation model according to the loss function to obtain a target quality evaluation model; inputting the image to be evaluated into the target quality evaluation model for calculation, and outputting the quality score of the image to be evaluated. The present invention utilizes the weak correlation between semantic segmentation and the BIQA task to enhance the representation learning ability of quality-aware features and improve the accuracy of image quality evaluation.

[0167] It should be noted that in this article, the term "including", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or terminal including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of another identical element in the process, method, article or terminal including the element.

[0168] Of course, those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the various embodiments provided by the present invention can include non-volatile and / or volatile memories. Non-volatile memories can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memories can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0169] It should be understood that the application of the present invention is not limited to the above examples. For those of ordinary skill in the art, improvements or transformations can be made according to the above description. All such improvements and transformations should fall within the protection scope of the appended claims of the present invention.

Claims

1. A reference - free image quality assessment method based on weakly - correlated knowledge distillation, characterized in that, The described no-reference image quality assessment method based on weak-correlation knowledge distillation includes: Input the distorted images in the training samples into a quality assessment model and a semantic segmentation model respectively for feature extraction, obtaining multi-scale quality features and multi-scale semantic features; Perform correlation constraints on the multi-scale quality features and the multi-scale semantic features, and calculate the correlation constraint loss; Fuse the multi-scale quality features and the multi-scale semantic features to obtain fused features, and perform quality regression on the fused features to obtain a predicted score; Calculate the overall loss function according to the predicted score and the correlation constraint loss, and update the weights of the quality assessment model according to the loss function to obtain the target quality assessment model; Input the image to be evaluated into the target quality assessment model for calculation, and output the quality score of the image to be evaluated.

2. The no-reference image quality assessment method based on weakly correlated knowledge distillation according to claim 1, wherein The backbone network types of the quality assessment model and the semantic segmentation model are the same to maintain the dimension alignment of feature representations.

3. The no-reference image quality assessment method based on weakly correlated knowledge distillation according to claim 1, wherein The performing correlation constraints on the multi-scale quality features and the multi-scale semantic features, and calculating the correlation constraint loss specifically includes: Calculate the first covariance matrix of the multi-scale quality features and the second covariance matrix of the multi-scale semantic features respectively, and calculate the difference matrix between the first covariance matrix and the second covariance matrix of the multi-scale semantic features; Deploy a clustering algorithm to the difference matrix, cluster and group the elements on the difference matrix to obtain a grouping result, and generate a first mask matrix and a second mask matrix according to the grouping result; Calculate the whitening loss according to the first mask matrix and the first covariance matrix, calculate the normalization loss according to the second mask matrix, and calculate the correlation constraint loss according to the whitening loss and the normalization loss.

4. The method for no-reference image quality assessment based on weakly correlated knowledge distillation according to claim 3, wherein The calculating the first covariance matrix of the multi-scale quality features and the second covariance matrix of the multi-scale semantic features respectively, and calculating the difference matrix between the first covariance matrix and the second covariance matrix of the multi-scale semantic features specifically includes: Flatten the multi-scale quality features and the multi-scale semantic features to obtain the flattened quality features and the flattened semantic features; Calculate the first covariance matrix of the flattened quality features and the second covariance matrix of the flattened semantic features: Among them, is the first covariance matrix, is the second covariance matrix, h and w are the height and width of the corresponding feature respectively, and F q ' is the flattened mass feature, and F s ' is the flattened semantic feature, and T is the transpose operation; Calculate the difference matrix between the first covariance matrix and the second covariance matrix of the multi-scale semantic features according to the upper triangular matrices of the first covariance matrix and the second covariance matrix: where D is the difference matrix, || + denotes taking the upper triangular part of the difference matrix.

5. The method for no-reference image quality assessment based on weakly correlated knowledge distillation according to claim 4, wherein The deploying a clustering algorithm to the difference matrix, clustering and grouping the elements on the difference matrix to obtain a grouping result, and generating a first mask matrix and a second mask matrix according to the grouping result specifically includes: Deploy the k-means clustering algorithm to the difference matrix, cluster the elements on the difference matrix into k clusters, and divide the k clusters into a low-difference value group and a high-difference value group according to a preset threshold; Generate a first mask matrix and a second mask matrix according to the low-difference value group and the high-difference value group: Among them, M l is the first mask matrix, M h is the second mask matrix, D x,y represents the element at the x - coordinate and y - coordinate in the difference matrix, G low is the low - difference value group, G high is the high - difference value group.

6. The method for no-reference image quality assessment based on weakly correlated knowledge distillation according to claim 5, wherein The whitening loss is calculated based on the first mask matrix and the first covariance matrix, the normalization loss is calculated based on the second mask matrix, and the correlation constraint loss is calculated based on the whitening loss and the normalization loss. Specifically, it includes: Multiply the first mask matrix by the first covariance matrix to calculate the whitening loss to suppress covariance terms with low difference values: Calculate the normalization loss based on the second mask matrix to expand the feature distance: L h = E[|D⊙M h -M h |1]; Add the whitening loss and the normalization loss to obtain the correlation constraint loss: L cc = L l + L h ; Among them, L l is the whitening loss, L h is the normalization loss, L cc is the correlation constraint loss, ⊙ is the Hadamard product, ||1 means to accumulate after taking the absolute value of the elements therein, and E is the expectation.

7. The no-reference image quality assessment method based on weakly correlated knowledge distillation according to claim 6, wherein The calculation of the overall loss function based on the prediction score and the correlation constraint loss specifically includes: Obtain the quality evaluation task loss based on the prediction score and the average subjective score corresponding to the prediction score; Calculate the overall loss function based on the quality evaluation task loss and the correlation constraint loss: Among them, L total is the overall loss function, N is the batch size, M is all the feature layers adding the correlation constraint loss, i is the ordinal number of the sample, j corresponds to the ordinal number of the feature layer, and λ is the weight of the correlation constraint loss. is the loss of the quality evaluation task for the i-th sample. The correlation constraint loss of the j-th feature layer of the i-th sample.

8. A reference-free image quality assessment system based on weakly correlated knowledge distillation, characterized in that The no-reference image quality evaluation system based on weak correlation knowledge distillation includes: A multi-scale feature extraction module, which is used to input the distorted images in the training samples into a quality evaluation model and a semantic segmentation model respectively for feature extraction to obtain multi-scale quality features and multi-scale semantic features; A multi-scale correlation constraint module, which is used to perform correlation constraints on the multi-scale quality features and the multi-scale semantic features and calculate the correlation constraint loss; A feature fusion regression module, which is used to fuse the multi-scale quality features and the multi-scale semantic features to obtain fused features, and perform quality regression on the fused features to obtain prediction scores; A quality evaluation model update module, which is used to calculate the overall loss function based on the prediction score and the correlation constraint loss, and update the weights of the quality evaluation model according to the loss function to obtain the target quality evaluation model; A quality score prediction module, which is used to input the image to be evaluated into the target quality evaluation model for calculation and output the quality score of the image to be evaluated.

9. A terminal, characterized in that, The terminal includes: a memory, a processor, and a no-reference image quality evaluation program based on weak correlation knowledge distillation stored on the memory and executable on the processor. When the no-reference image quality evaluation program based on weak correlation knowledge distillation is executed by the processor, the steps of the no-reference image quality evaluation method according to any one of claims 1-7 are implemented.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a no-reference image quality evaluation program based on weak correlation knowledge distillation. When the no-reference image quality evaluation program based on weak correlation knowledge distillation is executed by a processor, the steps of the no-reference image quality evaluation method according to any one of claims 1-7 are implemented.

Images