No-reference image quality assessment method based on causal distillation

Abstract

This paper proposes a no-reference image quality assessment method based on causal distillation. The implementation steps are as follows: obtaining a training sample set and a test sample set; constructing a no-reference image quality assessment network model based on causal distillation; iteratively training the no-reference image quality assessment network model based on causal distillation; and obtaining no-reference image quality assessment results. The present invention constructs a causal distillation subnetwork based on the causal distillation strategy, adjusting feature weights during training. This effectively avoids the catastrophic forgetting problem caused by distillation loss in transfer learning, which is common in existing techniques. This method ensures good prediction accuracy for multiple datasets, effectively improving the accuracy of no-reference image quality assessment.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology and relates to a no-reference image quality assessment method, specifically a no-reference image quality assessment method based on causal distillation. This method can be used in technical fields such as image processing and enhancement, video coding and transmission, image retrieval and indexing, visual perception technology evaluation, and video surveillance and security. It can guide image quality processing and enhancement, optimize video coding algorithms, select optimal transmission strategies, and evaluate and improve the performance of visual perception technologies. Background Art

[0002] Images play a critical role in every field of modern society, from media and communication to medicine, engineering, art, and scientific research. Images help convey information, capture memories, inspire creativity, and evoke emotional resonance. However, the impact of an image often depends on its quality. If an image is of poor quality—with blurred pixels, distorted colors, or unclear details—it may not accurately convey information or produce the desired effect. For example, in medicine, a poor-quality image can lead to a misdiagnosis or miss important details. Similarly, in advertising and media, low-quality images may fail to capture audience interest or inspire purchases.

[0003] Therefore, the utility of image quality assessment has become particularly important. By objectively and comprehensively evaluating image quality, we can determine the reliability, accuracy, and usability of images. This evaluation can help us determine whether an image is suitable for a specific application scenario, thereby improving the efficiency and accuracy of information transmission. Objective image quality assessment, with its objectivity, time-saving, and low-cost advantages, has become a research focus in the field of image quality assessment.

[0004] Objective image quality assessment involves the process of objectively scoring images through the establishment of mathematical models. By simulating the human visual system's perception of images, an objective evaluation model capable of reflecting subjective image quality is designed to assess the quality of the image being tested. With the advancement of machine learning and deep learning, objective image quality assessment methods are also evolving and improving. By leveraging technologies such as large-scale image databases and deep neural networks, more accurate and sophisticated objective evaluation models can be constructed, further improving the utility and reliability of image quality assessment.

[0005] Objective image quality assessment methods can be categorized as full-reference, semi-reference, and no-reference methods based on their reliance on reference images. No-reference methods, which do not require a reference image, have become a hot topic and a key focus of current research. In practical applications, we often face the challenge of not having access to the original reference image, necessitating a method that can accurately assess image quality without a reference image. This situation is particularly common in areas such as image transmission, storage, and processing. Therefore, no-reference image quality assessment has become a key research focus in the field of image quality assessment.

[0006] The goal of no-reference image quality assessment is to accurately predict image quality by analyzing the image's features and content. This requires extracting features related to human subjective perception from the image and establishing a mapping relationship between these features and the subjective evaluation results. This method does not rely on a reference image and can accurately assess images in the absence of reference information, making it of great practical value. Research on no-reference image quality assessment involves the intersection of multiple fields, including computer vision, image processing, and machine learning, and requires full consideration of factors such as the image's structure, content, distortion type, and the perceptual mechanisms of the human visual system. Through in-depth research and exploration, no-reference image quality assessment methods have continuously achieved breakthroughs and progress, providing new approaches and solutions to image quality assessment problems in practical applications.

[0007] Existing no-reference image quality assessments are insufficiently accurate. Guanghao Yin et al. published a paper titled "Content-Variant Reference Image Quality Assessment via Knowledge Distillation" in the 3rd article of Volume 36 of the 2022 Proceedings of the AAAI Conference on Artificial Intelligence. This method, for the first time, attempts to transfer more prior knowledge of high-quality-low-quality differences from full-reference image quality assessment to no-reference image quality assessment through knowledge distillation. However, this method uses distillation loss to transfer knowledge. Due to the significant differences between different real distorted image datasets, the distillation loss will mislead new training, resulting in insufficient prediction accuracy for the no-reference image quality assessment model. Summary of the Invention

[0008] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and propose a reference-free image quality evaluation method based on causal distillation to solve the technical problem of low evaluation accuracy in the prior art.

[0009] To achieve the above object, the technical method adopted by the present invention includes the following steps:

[0010] (1) Obtain training sample set and test sample set:

[0011] Get L RGB images of dimensions W×H×C and each image I l Subjective quality score labels s l And the M RGB images and their corresponding subjective quality score labels form a training sample set T M ={(I m , s m )|1≤m≤M}, the remaining N RGB images and their corresponding subjective quality score labels form the test sample set T N ={(I n , s n )|1≤n≤N}, where L≥1000, W, H, and C represent the width, height, and number of channels of the RGB image, respectively. W≥224, H≥224, and C≥3. M+N=L;

[0012] (2) Construct a no-reference image quality assessment network model G based on causal distillation:

[0013] Construct a no-reference image quality assessment network model G that includes a sequentially connected image local feature extraction subnetwork R, a causal distillation subnetwork D, an image local feature global association subnetwork T, and an image quality prediction subnetwork P, and whose loss function is Loss; wherein R contains sequentially connected convolutional layers, pooling layers, and multiple feature extraction units U, U contains sequentially connected multiple convolutional layers and adaptive average pooling layers; D contains multiple MLP mixers M arranged in parallel with the same number of feature extraction units U, M contains multiple classic Vision MLP units; T contains sequentially connected feature splicing layers and global pooling layers; P contains multiple stacked fully connected layers; wherein the expression of Loss is:

[0014]

[0015]

[0016] L1=|s i -f(I i |G)|

[0017] L2=max(0,-(s i -s j )·(f(I i |G)-f(I j |G)))

[0018] Among them, B represents the number of training samples selected for each training, I i and I j Represents the i-th and j-th RGB images in B, s i and s j Indicates I i and I j The corresponding subjective quality score label, f(I i |G) and f(I j |G) indicates I i and I j The prediction quality score of

[0019] (3) Initialization parameters:

[0020] The initial iteration number is k, the maximum iteration number is K, K>20, the current network model G k The weight parameter is θ k ;

[0021] (4) Training the no-reference image quality assessment network model based on causal distillation:

[0022] From the training sample set T M B training samples randomly selected without replacement are used as the current no-reference image quality evaluation network model G k The forward reasoning of the input includes the following steps:

[0023] The image local feature extraction subnetwork R performs Q feature extractions on each training sample to obtain Q deep features F for each training sample. b ;Causal distillation sub-network D for down-sampled features F b Perform causal distillation to obtain Q causal distillation features H containing causal information b ; Image local feature global correlation sub-network T to H b The Q causal distillation features are concatenated and globally pooled to obtain the breadth perception feature E corresponding to each training sample, which contains the local-global correlation information and causal intervention information of the image. b ; Image quality prediction subnetwork P vs. E b Perform linear regression to obtain the quality prediction score S of each training sample b , where B≥16,Q≥2;

[0024] (5) Update the weight parameters of the network model:

[0025] The quality prediction score S of each training sample b For the network model G k The weight parameter θ k Update and get the network model G of this iterationk ;

[0026] (6) Obtain the trained no-reference image quality assessment network model G based on causal distillation * :

[0027] Determine whether k=K is true. If so, obtain the trained no-reference image quality evaluation network model G based on causal distillation * , otherwise, let k = k + 1 and execute step (4);

[0028] (7) Obtaining the quality evaluation results of the non-reference image:

[0029] The test sample set T N As the trained causal distillation-based no-reference image quality assessment network model G * The input is used for forward reasoning to obtain the quality prediction score S of each test sample N .

[0030] Compared with the prior art, the present invention has the following advantages:

[0031] 1. The no-reference image quality assessment network model constructed in this paper includes a causal distillation sub-network. During the model training and acquisition of no-reference image quality assessment results, the causal distillation sub-network dynamically adjusts the weights of deep features, solving the catastrophic forgetting problem faced by the algorithm in transfer learning and improving the image quality assessment prediction accuracy.

[0032] 2. Experiments show that the present invention can achieve higher prediction accuracy than the existing technology in real distorted image evaluation datasets. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is an implementation flow chart of the present invention.

[0034] Figure 2 It is a structural diagram of the reference-free image quality assessment network model based on causal distillation constructed by the present invention. DETAILED DESCRIPTION

[0035] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0036] Reference Figure 1 , the present invention comprises the following steps:

[0037] Step 1) Obtain training sample set and test sample set:

[0038] Get L RGB images of dimensions W×H×C and the subjective quality score label s of each image lAnd the M RGB images and their corresponding subjective quality score labels form a training sample set T M ={(I m ,s m )|1≤m≤M}, the remaining N RGB images and their corresponding subjective quality score labels form the test sample set T N ={(I n ,s n )|1≤n≤N}, where L≥1000, W, H, and C represent the width, height, and number of channels of the RGB image, respectively. W≥224, H≥224, and C≥3. M+N=L,I l Represents the lth RGB image;

[0039] In this example, W=224, H=224, C=3, and L=1000.

[0040] Step 2) Construct a no-reference image quality assessment network model G based on causal distillation:

[0041] (2a) Construct a reference-free image quality assessment network model G comprising a sequentially connected image local feature extraction subnetwork R, a causal distillation subnetwork D, an image local feature global association subnetwork T, and an image quality prediction subnetwork P, with a loss function of Loss; wherein R comprises a sequentially connected convolutional layer, a pooling layer, and a plurality of feature extraction units U, U comprises a sequentially connected plurality of convolutional layers and an adaptive average pooling layer; D comprises a plurality of MLP mixers M arranged in parallel with the same number of feature extraction units U, M comprises a plurality of classic Vision MLP units; T comprises a sequentially connected feature splicing layer and a global pooling layer; P comprises a plurality of stacked fully connected layers;

[0042] In this example, the image local feature extraction subnetwork R contains a convolutional layer with a convolution kernel size of 7×7, a number of 64, and a stride of 2; the pooling layer contains a pool size of 3×3, and a stride of 2; the number of feature extraction units U is 4, denoted as U1, U2, U3, and U4. The number of convolutional layers contained in U1 is 9, and the sizes of the convolution kernels are 1×1, 3×3, 1×1, 1×1, 3×3, 1×1, 1×1, 3×3, and 1×1, respectively. The number is 64, 64, 256, 64, 64, 256, 64, 64, 256, with a step size of 2; U2 contains 12 convolutional layers, and the sizes of the convolution kernels are 1×1, 3×3, 1×1, 1×1, 3×3, 1×1, 1×1, 3×3, 1×1, 1×1, 3×3, 1×1, and the number is 128, 128, 512, 128, 128, 512, 128, 128, 512, 128, 128, 512, with a stride of 2; U3 contains 18 convolutional layers, and the sizes of the convolution kernels are 1×1, 3×3, 1×1, 1×1, 3×3, 1×1, 1×1, 3×3, 1×1, 1×1, 3×3, 1×1, 1×1, 3×3, 1×1, 1×1, 3×3, 1×1, 1×1, 3×3, 1×1, and the number is 256, 256, 1024, 256, 256, 1024, 256, 256, 1024, 256 , 256, 1024, 256, 256, 1024, 256, 256, 1024, with a step size of 2; the feature extraction subunit U4 contains 9 convolutional layers, and the sizes of the convolution kernels are 1×1, 3×3, 1×1, 1×1, 3×3, 1×1, 1×1, 3×3, 1×1, and the numbers are 512, 512, 2048, 512, 512, 2048, 512, 512, 2048, with a step size of 2;

[0043] The causal distillation subnetwork D, where each MLP mixer M contains 3 classic Vision MLP units;

[0044] The image quality prediction subnetwork P contains 5 fully connected layers, and the lengths of the 5 fully connected layers are 256, 128, 64, 32, and 16 respectively;

[0045] (2b) Define the loss function Loss of the no-reference image quality assessment network model G based on causal distillation:

[0046]

[0047]

[0048] L1=|s i -f(I i |G)|

[0049] L2=max(0,-(s i -s j )·(f(I i |G)-f(I j |G)))

[0050] Among them, B represents the number of training samples selected for each training, I i and I j Represents the i-th and j-th RGB images in B, s i and s j Indicates I i and I j The corresponding subjective quality score label, f(I i |G) and f(I j |G) represents the model pair I i and I j The prediction quality score of

[0051] In this example, B = 200;

[0052] The loss function, Loss, consists of two loss terms: a quality prediction loss function and a ranking and distance loss function. During the training of the image quality assessment network model, a dynamic weighting coefficient λ dynamically assigns weights to the two loss terms. In the initial stages of training, the ranking and distance metric loss function has the highest weight. As the number of iterations increases, the weights of the ranking and distance metric loss functions gradually decrease, while the weight of the quality prediction loss function increases. This loss function structure can enhance the linear correlation between subjective quality scores and quality predictions, improving the accuracy of quality assessment.

[0053] Step 3) Train the no-reference image quality assessment network model G based on causal distillation:

[0054] From the training sample set T M B training samples randomly selected without replacement are used as the current no-reference image quality evaluation network model G k The input is forward inferenced, and the quality prediction score of each training sample obtained by forward inference is used to predict G k The weight parameter is θ k Update for a total of K times, K>20, and obtain the trained no-reference image quality evaluation network model G based on causal distillation * , where forward reasoning includes the following steps:

[0055] (1) The convolution layer in the image local feature extraction subnetwork R maps each training sample to a high dimension, and then the pooling layer downsamples the convolution operation result to obtain the initial feature F of each training sample b '; Q feature extraction units for Fb ′ performs feature extraction to obtain Q features of different depths F for each training sample b , where the output of the qth feature extraction unit is F bq .

[0056] (2) The MLP mixer M connected to each feature extraction unit in the causal distillation subnetwork D generates the feature F of each corresponding training sample. bq Perform causal distillation and obtain the same b The corresponding Q causal distillation features H containing causal information b :

[0057] H b =softmax(w2*(w1*F b +β1)+β2)

[0058] Among them, w1, w2 represent weight parameters, β1, β2 represent bias parameters, and softmax(·) represents the normalized exponential function.

[0059] (3) Image local feature global correlation sub-network T to H b The Q causal distillation features are concatenated and globally pooled to obtain the breadth perception feature E corresponding to each training sample, which contains the local-global correlation information and causal intervention information of the image. b ;

[0060] (4) Image quality prediction subnetwork P vs. E b Perform linear regression to obtain the quality prediction score S of each training sample b , where B ≥ 16, Q ≥ 2;

[0061] In this example, Q = 4, the loss function Loss is used, and the error between the quality prediction score of the training sample calculated by the model and the subjective quality score label corresponding to the training sample is used as G k The loss value d k ; then through d k Calculate the G k The weight parameter θ k The partial derivative g k , and then use the stochastic gradient descent method, through g k To G k The weight parameter θ k Update, where the partial derivative g k The calculation formula and weight parameter θ k The update formulas are:

[0062]

[0063]

[0064] in, Represents the no-reference image quality assessment network model G based on causal distillation k The updated weight parameters; α represents the learning rate, α = 1×10 -4 .

[0065] (4) Obtaining the quality evaluation results of the non-reference image:

[0066] The test sample set T N As the trained causal distillation-based no-reference image quality assessment network model G * The input is used for forward reasoning to obtain the quality prediction score S of each test sample N .

Claims

1. A no-reference image quality assessment method based on causal distillation, characterized in that: The steps include: (1) Obtain training sample set and test sample set: Get L RGB images of dimensions W×H×C and each image I l Subjective quality score labels s l And the M RGB images and their corresponding subjective quality score labels form a training sample set T M ={(I m ,s m )|1≤m≤M}, the remaining N RGB images and their corresponding subjective quality score labels form the test sample set T N ={(I n ,s n )|1≤n≤N}, where L≥1000, W, H, and C represent the width, height, and number of channels of the RGB image, respectively. W≥224, H≥224, and C≥3. M+N=L; (2) Construct a no-reference image quality assessment network model G based on causal distillation: Construct a no-reference image quality assessment network model G that includes a sequentially connected image local feature extraction subnetwork R, a causal distillation subnetwork D, an image local feature global association subnetwork T, and an image quality prediction subnetwork P, and whose loss function is Loss; wherein R contains sequentially connected convolutional layers, pooling layers, and multiple feature extraction units U, U contains sequentially connected multiple convolutional layers and adaptive average pooling layers; D contains multiple MLP mixers M arranged in parallel with the same number of feature extraction units U, M contains multiple classic Vision MLP units; T contains sequentially connected feature splicing layers and global pooling layers; P contains multiple stacked fully connected layers; wherein the expression of Loss is: L1=|s i -f(I i |G)| L2=max(0,-(s i -s j )·(f(I i |G)-f(I j |G))) Among them, B represents the number of training samples selected for each training, I i and I j Represents the i-th and j-th RGB images in B, s i and s j Indicates I i and I j The corresponding subjective quality score label, f(I i |G) and f(I j |G) indicates I i and I j The prediction quality score of (3) Initialization parameters: The initial iteration number is k, the maximum iteration number is K, K>20, the current network model G k The weight parameter is θ k ; (4) Training the no-reference image quality assessment network model based on causal distillation: From the training sample set T M B training samples randomly selected without replacement are used as the current no-reference image quality evaluation network model G k The forward reasoning of the input includes the following steps: The image local feature extraction subnetwork R performs Q feature extractions on each training sample to obtain Q deep features F for each training sample. b ;Causal distillation sub-network D for down-sampled features F b Perform causal distillation to obtain Q causal distillation features H containing causal information b ; Image local feature global correlation sub-network T to H b The Q causal distillation features are concatenated and globally pooled to obtain the breadth perception feature E corresponding to each training sample, which contains the local-global correlation information and causal intervention information of the image. b ; Image quality prediction subnetwork P vs. E b Perform linear regression to obtain the quality prediction score S of each training sample b , where B≥16,Q≥2; (5) Update the weight parameters of the network model: The quality prediction score S of each training sample b For the network model G k The weight parameter θ k Update and get the network model G of this iteration k ; (6) Obtain the trained no-reference image quality assessment network model G based on causal distillation * : Determine whether k=K is true. If so, obtain the trained no-reference image quality evaluation network model G based on causal distillation * , otherwise, let k = k + 1 and execute step (4); (7) Obtaining the quality evaluation results of the non-reference image: The test sample set T N As the trained causal distillation-based no-reference image quality assessment network model G * The input is used for forward reasoning to obtain the quality prediction score S of each test sample N .

2. The method according to claim 1, characterized in that The no-reference image quality assessment network model G described in step (2), wherein: The image local feature extraction subnetwork R contains a convolution layer with a convolution kernel size of 7×7, a number of 64, and a step size of 2; the pooling layer contains a pool size of 3×3, and a step size of 2; the number of feature extraction units U is 4, denoted as U1, U2, U3, and U4, and the number of convolution layers contained in U1 is 9, and the sizes of the convolution kernels are 1×1, 3×3, 1×1, 1×1, 3×3, 1×1, 1×1, 3×3, and 1×1, in order. The number of convolutional layers in U2 is 12, and the sizes of convolutional kernels are 1×1, 3×3, 1×1, 1×1, 3×3, 1×1, 1×1, 3×3, 1×1, 1×1, 3×3, 1×1, 1×1, 3×3, 1×1, and the number of kernels is 128, 128, 512, ... 8,512, with a stride of 2; U3 contains 18 convolutional layers, and the sizes of the convolution kernels are 1×1, 3×3, 1×1, 1×1, 3×3, 1×1, 1×1, 3×3, 1×1, 1×1, 3×3, 1×1, 1×1, 3×3, 1×1, 1×1, 3×3, 1×1, 1×1, 3×3, 1×1, and the number is 256, 256, 1024, 256, 256, 1024, 256, 256, 1024, 256, 256, 1024, 256, 256, 1024, 256, 256, 1024, with a step size of 2; the feature extraction subunit U4 contains 9 convolutional layers, and the sizes of the convolution kernels are 1×1, 3×3, 1×1, 1×1, 3×3, 1×1, 1×1, 3×3, 1×1, and the numbers are 512, 512, 2048, 512, 512, 2048, 512, 512, 2048, with a step size of 2; The causal distillation subnetwork D, where each MLP mixer M contains 3 classic VisionMLP units; The image quality prediction subnetwork P contains 5 fully connected layers, and the lengths of the 5 fully connected layers are 256, 128, 64, 32, and 16 respectively.

3. The method according to claim 1, characterized in that The image local feature extraction subnetwork R described in step (4) performs Q feature extractions on each training sample, and the implementation steps are as follows: The convolution layer in the image local feature extraction subnetwork R maps each training sample to a high dimension, and then the pooling layer downsamples the convolution operation result to obtain the initial feature F of each training sample. b '; Q feature extraction units for F b ′ performs feature extraction to obtain Q features of different depths F for each training sample b , where the output of the qth feature extraction unit is F bq .

4. The method according to claim 1, wherein The causal distillation sub-network D described in step (4) is used to calculate the depth feature F b Perform causal distillation and implement the steps as follows: The MLP mixer M connected to each feature extraction unit in the causal distillation subnetwork D generates the feature F of each corresponding training sample. bq Perform causal distillation and obtain the same b The corresponding Q causal distillation features H containing causal information b : H b =softmax(w2*(w1*F b +β1)+β2) Among them, w1, w2 represent weight parameters, β1, β2 represent bias parameters, and softmax(·) represents the normalized exponential function.

5. The method according to claim 1, wherein The breadth perception feature E described in step (4) b , the acquisition steps are: The feature concatenation layer in the global correlation sub-network T of the local image features sequentially processes the Q causal distillation features H according to the order of multiple feature extraction units. b Splicing; the global pooling layer splices the fusion feature E b ′ performs global pooling to obtain the breadth perception feature E corresponding to each training sample, which contains the local-global correlation information and causal intervention information of the image b .

6. The method according to claim 1, characterized in that The quality prediction score of each training sample in step (5) is used to evaluate the quality of the no-reference image network model G. k To update, the steps are: Use the loss function Loss and predict the quality score S of each training sample b and its corresponding subjective quality score label s b Calculate G k The loss value d k ; then through d k Calculate the G k The weight parameter θ k The partial derivative g k , and then use the stochastic gradient descent method, through g k To G k The weight parameter θ k Update, where the partial derivative g k The calculation formula and weight parameter θ k The update formulas are: in, Represents the no-reference image quality assessment network model G based on curriculum learning k The updated weight parameters; α represents the learning rate.

Images