Real distortion full-reference image quality evaluation method and system and medium
By combining Res2net and Swin Transformer networks, using significance detection and channel and spatial attention mechanisms to extract image features, the problem of insufficient global information capture and feature representation in image quality evaluation in the prior art is solved, and the human eye is realized Perceptually consistent high-accuracy image quality evaluation.
Patent Information
- Application Number
- CN202411103153.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-13
- Publication Date
- 2024-11-22
AI Technical Summary
The existing full reference image quality evaluation method has limitations in capturing global information and quality-perceptual features of the image. In particular, the fixed receptive field of the convolutional neural network limits the capture ability of the model, and deep learning methods cannot effectively capture long-distance features. Existing Transformer-based methods also fail to make full use of channel and spatial attention, resulting in inaccurate image quality evaluation.
The Res2net network and Swin Transformer network combine significance detection, channel and space dual attention mechanisms, and the image features are extracted through distortion perception and significance map detection module, and the channel and space attention modules are used to enhance feature representation, and finally, through the full range of Connection layers predict image quality scores.
It realizes a comprehensive and objective evaluation of image quality, can be consistent with human eye perception, and improves the accuracy of image quality evaluation and subtle distortion detection capabilities.
Smart Images

Figure CN119006988A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to a method, system and medium for evaluating the quality of a true-distortion full-reference image. Background Art
[0002] Images are an important part of modern media interaction. Low-quality images not only affect the user experience, but also lead to deviations and misunderstandings in information communication. Blurred or distorted images affect the understanding and perception of image content. Therefore, maintaining high image quality is crucial to ensure accurate transmission of information and correct understanding by users.
[0003] The full-reference method of image quality can use the complete information of the reference image for precise comparison and analysis, thereby providing more accurate and reliable quality evaluation results. This is of great significance for application scenarios that require high-precision image quality assurance, such as medical imaging, satellite remote sensing, digital image restoration, etc. Secondly, the full-reference method can better detect and evaluate subtle distortions and detail loss in the image, thereby providing more comprehensive image quality feedback. Most of the existing full-reference methods are based on deep learning, but deep learning uses convolutional neural networks to extract quality-perceived features of images. It cannot capture long-distance features and requires deepening the number of network layers to obtain a larger receptive field.
[0004] Transformer has performed very well in the field of computer vision. Among them, Swin Transformer introduced the window attention mechanism (Window-based Multi-Head Self-Attention), which divides the image into multiple windows of fixed size and calculates the attention independently in each window. This method greatly reduces the computational complexity and shows great application prospects. Swin Transformer can use the local attention mechanism to process large-size images, and also has the advantage of the sliding window solution to establish long-distance dependencies between features.
[0005] However, the convolution operation in the convolutional neural network (CNN) has a fixed-size receptive field, which limits the model's ability to capture global information. The existing Transformer-based methods also still have room for improvement. For example, the features of all channels are treated equally and are not adjusted according to their importance. That is, the features of different stages of the deep network should be considered to different degrees. In addition, how to better fit the quality-aware features to the predicted scores is a difficult problem that urgently needs to be solved by technology. Summary of the invention
[0006] The purpose of this application is to provide a method, system and medium for evaluating the quality of a real distorted full-reference image, which can use the Res2net network and the Swin Transformer network to perform saliency detection, channel and spatial dual attention and distortion residual map processing, and make a similar subjective perception consistency evaluation of image quality.
[0007] The present application also provides a method for evaluating the quality of a true-distortion full-reference image, comprising the following steps:
[0008] Obtaining a full reference image quality assessment dataset including a training image set with real distortion, and generating an image data set;
[0009] Obtain the initial full-reference image quality assessment model, including the distortion-aware reconstruction network module, the saliency map detection module, the local and global feature extraction network modules, and the channel and spatial attention modules;
[0010] Preprocessing the training image set in the image data set and inputting the training image set into the full-reference image quality initial evaluation model for training to obtain a trained full-reference image quality evaluation model;
[0011] The real distorted image set to be evaluated is input into the trained full-reference image quality evaluation model for processing to obtain an image quality score.
[0012] Optionally, in the real distortion full-reference image quality assessment method described in the present application, the distortion-aware reconstruction network module includes:
[0013] The distortion-aware reconstruction network module includes an encoder structure, a decoder structure and a Short-cut layer;
[0014] The encoder structure includes a plurality of sub-layers, including a plurality of convolutional layers and activation functions, and extracts features of the distorted image according to the encoder structure;
[0015] The calculation formula for extracting distorted image features by the encoder structure is:
[0016] F = ReLU(Conv*I d ();
[0017] Among them, F is the degradation graph feature extracted by the encoder structure, I d is the degradation graph, Conv(·) is the convolutional layer, and ReLU(·) is the activation function;
[0018] The decoder structure comprises a plurality of sub-layers, including a plurality of deconvolution layers and activation functions, and extracts degradation map features according to the encoder structure and reconstructs and generates a corresponding distortion residual map;
[0019] The calculation formula of the distortion residual map reconstructed by the decoder structure is:
[0020] E=ReLU(ConvTranspose2d*F));
[0021] Among them, E is the distortion residual map reconstructed by the decoder structure, F is the degradation map feature extracted by the encoder structure, ConvTranspose2d(·) is the deconvolution layer, and ReLU(·) is the activation function;
[0022] The Short-cut layer obtains a distortion residual map according to the reference image and the distorted image, and then obtains a picture by comparing and splicing the distortion residual map and the degradation map, and performs difference comparison with the reference image to train the distortion-aware reconstruction network module;
[0023] The distortion-aware reconstruction network module also includes an error-aware reconstruction network model, which includes a mean absolute error and a structural similarity index;
[0024] The function calculation formula of the mean absolute error is:
[0025]
[0026] in, is the loss function of mean absolute error, |x i -y i | is the absolute value of the difference between the i-th corresponding pixels of the two images;
[0027] The functional calculation formula of the structural similarity index is:
[0028]
[0029] in, is the loss function of the structural similarity index, μ x and μ y are the average values of the contrast features of the two images x and y, σ x and σ y are the standard deviations of the contrast features of the two images x and y, σ xy is the contrast feature covariance of x and y, c1 and c2 are the preset feature constant coefficients;
[0030] Performing objective function minimization processing on the mean absolute error and the structural similarity index according to the error-aware reconstruction network model;
[0031] The calculation formula for minimizing the objective function of the error-aware reconstruction network is:
[0032]
[0033] Among them, Id is the degradation graph, E is the distortion residual graph reconstructed by the decoder structure, and I ref is the corresponding reference graph, and λ is the preset weighting coefficient.
[0034] Optionally, in the real distortion full-reference image quality assessment method described in the present application, the saliency map detection module includes:
[0035] The saliency map detection module calculates the gradient of the image in the x-axis and y-axis directions through convolution operation, and the calculation formula is:
[0036]
[0037] Among them, P x and P y are the gradients of the image in the x-axis and y-axis directions, h x and h y are the convolution kernels along the x-axis and y-axis respectively, is the convolution operator, s is the image signal;
[0038] The gradient amplitude of the image is calculated according to the gradients in the x-axis and y-axis directions, and the calculation formula is:
[0039]
[0040] Where P is the gradient amplitude;
[0041] Then perform Gaussian low-pass filtering on the image to obtain a smoothed image. The calculation formula is:
[0042]
[0043] Among them, r g is the smoothed image, r is the original image, and g represents the Gaussian filter;
[0044] The function formula of Gaussian filter is:
[0045]
[0046] Then, the saliency map is processed by combining the gradient amplitude of the original image and the gradient amplitude of the image smoothed by Gaussian low-pass filtering to obtain a saliency image corresponding to the original image;
[0047] The processing calculation formula of the saliency image is:
[0048]
[0049] Among them, M(r) is the saliency image, P r is the gradient amplitude of the original image, is the gradient amplitude of the smoothed image, C is a constant, Where L is the dynamic range of pixel values, and K is a scalar constant;
[0050] Performing enhancement processing on the original image according to the saliency image to obtain an enhanced image;
[0051] The processing calculation formula of the enhanced image is:
[0052] I enhanced =I original ·(1+k·M);
[0053] Among them, I enhanced is the enhanced image, I original is the original image, k is the preset enhancement coefficient, and M is the saliency map.
[0054] Optionally, in the real distortion full-reference image quality assessment method described in the present application, the local and global feature extraction network module includes:
[0055] The local and global feature extraction network modules include Res2Net and SwinTransformer:
[0056] According to Res2Net, feature X is divided into s scales, expressed as The features of each scale are represented by X i ;
[0057] Y i is the output feature of the i-th scale in the s-th scale, Y1=X1,Y i =Conv i (X i +Y i-1 );
[0058] Among them, Conv i is the convolution function of the i-th scale, i = 2, 3, ...s;
[0059] Perform splicing processing according to the features of each size and combine them with preset residuals to obtain final output features;
[0060] The calculation formula of the final output feature is:
[0061]
[0062] Among them, Z is the final output feature, and X is the preset residual;
[0063] Perform attention calculation on the feature X according to Swin Transformer, including:
[0064] Divide the feature X into windows:
[0065] X=Partition(X,window_size);
[0066] Perform self-attention calculation and linear transformation on the features in each divided window:
[0067]
[0068] Y=Attention(Q,K,V)W o ;
[0069] Among them, Attention(Q,K,V) is the self-attention calculation function, Y is the linear transformation of the window feature, Q, K, V are query, key and value respectively, d k is the dimension of the key, W o is the linear transformation matrix;
[0070] Merge the features in each window to obtain the global features:
[0071] Z = Merge(Y);
[0072] Among them, Z is the global feature, Merge(·) is the merging function;
[0073] The initial features are extracted by two shallow convolution blocks, and we get and And input into the Res2Net stream and SwinTransformer stream for processing respectively, and output CNN features and Transformer features respectively;
[0074] The output formula of the CNN feature and Transformer feature is:
[0075]
[0076] Among them, C(·) and T(·) are the encoder blocks of Res2Net and Swin Transformer respectively. and are the CNN features and Transformer features output at stage s∈{1,2,3,…,n}, respectively, Z s It is the feature obtained by multiplying the distorted residual map and the distorted image through s steps of convolution and downsampling.
[0077] Optionally, in the real distortion full-reference image quality assessment method described in the present application, the channel and spatial attention module includes:
[0078] The channel and spatial attention module includes a channel attention module CAM and a spatial attention module SAM;
[0079] Get feature image And perform global average pooling and global maximum pooling respectively to obtain the corresponding feature descriptions:
[0080]
[0081] Describe the characteristics Input the preset shared two-layer perceptron for processing respectively, and obtain the channel attention features respectively:
[0082]
[0083] Among them, W0 and W1 are weight matrices, σ is the sigmoid function, r is the preset compression ratio coefficient;
[0084] The channel attention feature Through the sigmoid function processing, the channel attention weight coefficient is obtained:
[0085]
[0086] The channel attention weight coefficient M c (F) Processing with the feature image to obtain the weighted channel attention feature:
[0087] F channel =M c (F)⊙F;
[0088] Attention feature of the weight channel Perform global average pooling and global maximum pooling respectively to obtain the corresponding feature images:
[0089]
[0090] According to the feature image Perform stitching processing to obtain the stitching feature image:
[0091]
[0092] The spliced feature image F concat The spatial attention weight coefficient is obtained by processing with the sigmoid function:
[0093] M s (F channel )=σ(Conv(F concat ));
[0094] Among them, Conv is the convolution function, σ is the sigmoid function;
[0095] The spatial attention weight coefficient M s (F channel ) is processed with the feature image to obtain the feature image after module enhancement processing:
[0096] F spatial =M s (F channel )⊙F channel ;
[0097] Outputting the feature image after the module enhancement processing as a module output feature image;
[0098] F out =F spatial ;
[0099] Among them, F out is the module output feature image, F spatial is the feature image after module enhancement, and ⊙ is the Hadamard product.
[0100] Optionally, in the real distortion full-reference image quality assessment method described in the present application, the channel and spatial attention module further includes:
[0101] According to the output feature image of the module, regression processing is performed through a preset fully connected layer to obtain an objective score;
[0102] F fc1 =ReLU(W1F out +b1);
[0103] Score objective =W2F fc1 +b2;
[0104] Among them, F fc1 is the output of the first fully connected layer, W1 is the weight matrix of the first fully connected layer, b1 is the preset bias vector of the first fully connected layer, ReLU(·) is the activation function, Score objective is the objective score of the second fully connected layer, W2 is the weight matrix of the second fully connected layer, b2 is the preset bias vector of the second fully connected layer,
[0105] Mapping the objective scores according to a nonlinear fitting model with preset parameters to obtain subjective scores;
[0106]
[0107] Among them, Score subjective is the subjective score, η i are preset fitting parameters, i=1,2,3,4,5.
[0108] Optionally, in the real distorted full-reference image quality assessment method described in the present application, inputting the real distorted image set to be evaluated into the trained full-reference image quality assessment model for processing to obtain an image quality score includes:
[0109] Acquire multiple image slices in a set of real distorted images to be evaluated;
[0110] Inputting the image slice into the trained full-reference image quality assessment model for processing to obtain a corresponding image prediction score value;
[0111] Processing the image slices according to the image prediction score values corresponding to the plurality of image slices to obtain an image quality score;
[0112] The calculation formula of the image quality score is:
[0113]
[0114] Among them, Score is the image quality score, n is the number of image slices, Pred j is the image prediction score corresponding to the j-th image slice.
[0115] In a second aspect, the present application provides a real distortion full reference image quality assessment system, the system comprising: a memory and a processor, the memory comprising a program of a real distortion full reference image quality assessment method, and the program of the real distortion full reference image quality assessment method is executed by the processor to implement the following steps:
[0116] Obtaining a full reference image quality assessment dataset including a training image set with real distortion, and generating an image data set;
[0117] Obtain the initial full-reference image quality assessment model, including the distortion-aware reconstruction network module, the saliency map detection module, the local and global feature extraction network modules, and the channel and spatial attention modules;
[0118] Preprocessing the training image set in the image data set and inputting the training image set into the full-reference image quality initial evaluation model for training to obtain a trained full-reference image quality evaluation model;
[0119] The real distorted image set to be evaluated is input into the trained full-reference image quality evaluation model for processing to obtain an image quality score.
[0120] Optionally, in the real distortion full-reference image quality assessment system described in the present application, the distortion-aware reconstruction network module includes:
[0121] The distortion-aware reconstruction network module includes an encoder structure, a decoder structure and a Short-cut layer;
[0122] The encoder structure includes a plurality of sub-layers, including a plurality of convolutional layers and activation functions, and extracts features of the distorted image according to the encoder structure;
[0123] The calculation formula for extracting distorted image features by the encoder structure is:
[0124] F = ReLU(Conv(I d ));
[0125] Among them, F is the degradation graph feature extracted by the encoder structure, I d is the degradation graph, Conv(·) is the convolutional layer, and ReLU(·) is the activation function;
[0126] The decoder structure comprises a plurality of sub-layers, including a plurality of deconvolution layers and activation functions, and extracts degradation map features according to the encoder structure and reconstructs and generates a corresponding distortion residual map;
[0127] The calculation formula of the distortion residual map reconstructed by the decoder structure is:
[0128] E=ReLU(ConvTranspose2d(F));
[0129] Among them, E is the distortion residual map reconstructed by the decoder structure, F is the degradation map feature extracted by the encoder structure, ConvTranspose2d(·) is the deconvolution layer, and ReLU(·) is the activation function;
[0130] The Short-cut layer obtains a distortion residual map according to the reference image and the distorted image, and then obtains a picture by comparing and splicing the distortion residual map and the degradation map, and performs difference comparison with the reference image to train the distortion-aware reconstruction network module;
[0131] The distortion-aware reconstruction network module also includes an error-aware reconstruction network model, which includes a mean absolute error and a structural similarity index;
[0132] The function calculation formula of the mean absolute error is:
[0133]
[0134] in, is the loss function of mean absolute error, |x i-y i | is the absolute value of the difference between the i-th corresponding pixels of the two images;
[0135] The functional calculation formula of the structural similarity index is:
[0136]
[0137] in, is the loss function of the structural similarity index, μ x and μ y are the average values of the contrast features of the two images x and y, σ x and σ y are the standard deviations of the contrast features of the two images x and y, σ xy is the contrast feature covariance of x and y, c1 and c2 are the preset feature constant coefficients;
[0138] Performing objective function minimization processing on the mean absolute error and the structural similarity index according to the error-aware reconstruction network model;
[0139] The calculation formula for minimizing the objective function of the error-aware reconstruction network is:
[0140]
[0141] Among them, I d is the degradation graph, E is the distortion residual graph reconstructed by the decoder structure, and I ref is the corresponding reference graph, and λ is the preset weighting coefficient.
[0142] In a third aspect, the present application also provides a computer-readable storage medium, which includes a real-distortion full-reference image quality assessment method program. When the real-distortion full-reference image quality assessment method program is executed by a processor, the steps of the real-distortion full-reference image quality assessment method as described in any one of the above items are implemented.
[0143] As can be seen from the above, the present application provides a method, system and medium for evaluating the quality of a real distorted full-reference image. The deep learning technology is used to interactively model the backbone network with Res2net and Swin Transformer as feature extraction to extract local and global features. The saliency map detection module is used to guide the model to focus on the area of human visual attention. At the same time, the distortion-aware reconstruction network is used to generate the corresponding distortion residual map to guide the attention calculation and perform residual connection to improve the performance of the network. The channel and spatial attention modules are connected to enhance the representation of features, and weights are allocated and learned. At the same time, the distortion residual map obtained by the distortion-aware reconstruction network is used to guide network learning. Then, two fully connected layers predict the score based on the quality perception features learned by the dual attention mechanism. Finally, the score is mapped to the subjective score through a five-parameter nonlinear fitting function to evaluate the image comprehensively and objectively, so that the model can make an evaluation of the image quality that is consistent with the subjective perception of the human eye.
[0144] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the embodiments of the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0145] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0146] Figure 1 A flowchart of a method for evaluating the quality of a true-distortion full-reference image provided in an embodiment of the present application. DETAILED DESCRIPTION
[0147] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.
[0148] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.
[0149] Please refer to Figure 1 , Figure 1 The flowchart of a real distortion full reference image quality evaluation method in some embodiments of the present application. The real distortion full reference image quality evaluation method is used in a terminal device, such as a computer, a mobile phone terminal, etc. The real distortion full reference image quality evaluation method comprises the following steps:
[0150] S1. Obtain a full reference image quality assessment dataset containing a training image set with real distortion, and generate an image data set;
[0151] In this embodiment, the dataset selected includes the full reference image quality assessment dataset LIVE. The LIVE dataset is the largest available annotated image quality dataset. The reference images of the entire dataset are 29 high-resolution and high-quality color images collected from the Internet and photography CDs, including face images, animal images, close-up images, wide-angle images, natural scenes, man-made objects, and images with different foreground / background configurations. The LIVE dataset includes 779 distorted images. The dataset uses five computer distortion operations to degrade the reference images by 5 to 6 levels, respectively, to obtain 779 distorted images. distorted images. In order to simulate most of the image damage that may occur in real applications, these distortion types include structured distortion (Gaussian blur), image-related distortion (JPEG compression, JPEG2000 compression and JPEG2000 fast scale fading distortion) and random noise (white noise), as well as 29 high-resolution 24-bit / pixel RGB color reference images. Each distortion type contains 5 or 4 distortion levels. The image size is mostly 768×512. The training set and test set are divided according to the 80-20 split, that is, 80% of the images are used for training and 20% for testing.
[0152] S2. Obtain the initial evaluation model of the full-reference image quality, including the distortion-aware reconstruction network module, the saliency map detection module, the local and global feature extraction network modules, and the channel and spatial attention modules;
[0153] The distortion-aware reconstruction network module described in this embodiment is a network based on the Unet architecture, including an encoder structure, a decoder structure and a Short-cut layer, and has distortion perception capability. The goal of the network is to reconstruct a distortion residual map between a distorted image and a reference image, wherein:
[0154] The encoder structure contains multiple sub-layers, each of which has the same structure: it contains multiple convolutional layers (Conv2d) and activation functions (ReLU) to extract the features of the distorted image. The encoder extracts the features as follows:
[0155] F = ReLU(Conv(I d ));
[0156] Among them, F is the degradation graph feature extracted by the encoder structure, I d is the degradation graph, Conv(·) is the convolutional layer, and ReLU(·) is the activation function;
[0157] The decoder structure contains multiple sub-layers, each of which has the same structure: it contains multiple deconvolution layers (ConvTranspose2d) and activation functions (ReLU), which are used to reconstruct the corresponding distorted residual map based on the features extracted from the encoder. The decoder structure reconstructs the distorted residual map, which can be expressed as:
[0158] E=ReLU(ConvTranspose2d(F));
[0159] Among them, E is the distortion residual map reconstructed by the decoder structure, F is the degradation map feature extracted by the encoder structure, ConvTranspose2d(·) is the deconvolution layer, and ReLU(·) is the activation function;
[0160] The short-cut layer is designed to guide the network to learn that the reconstruction target is not the reference image, but the distortion residual map between the reference image and the distorted image during the training process. The short-cut layer guides the network training by splicing the learned distortion residual map and degradation map and comparing the difference between the spliced image and the reference image.
[0161] The distortion-aware reconstruction network module also includes an error-aware reconstruction network model. The loss function of the error-aware reconstruction network model includes two items: Mean Absolute Error (MAE) and Structural Similarity Index (SSIM). MAE calculates the average of the absolute values of the corresponding pixel differences between two images, which is expressed as follows:
[0162]
[0163] in, is the loss function of mean absolute error, |x i -y i | is the absolute value of the difference between the i-th corresponding pixels of the two images;
[0164] SSIM measures the similarity of two images in terms of brightness, contrast, and structure. The value range of SSIM is [-1, 1], where 1 means they are exactly the same, as shown below:
[0165]
[0166] in, is the loss function of the structural similarity index, μ x and μ y are the average values of the contrast features of the two images x and y, σ x and σ y are the standard deviations of the contrast features of the two images x and y, σ xy is the contrast feature covariance of x and y, c1 and c2 are the preset feature constant coefficients;
[0167] The error-aware reconstruction network model performs objective function minimization processing on the mean absolute error and the structural similarity index, and its goal is to minimize the following objective function:
[0168]
[0169] Among them, I d is the degradation graph, E is the distortion residual graph reconstructed by the decoder structure, and I ref is the corresponding reference graph, and λ is the preset weighting coefficient.
[0170] The saliency map detection module described in this embodiment is used for saliency detection, and specifically includes:
[0171] Saliency detection refers to identifying and extracting salient areas that attract human visual attention from an image. Visual saliency can improve the performance of the image quality assessment module in practical applications. First, the gradient of the image in the x-axis and y-axis directions is calculated through convolution operations as follows:
[0172]
[0173] Among them, P x and P y are the gradients of the image in the x-axis and y-axis directions, h x and h y are the convolution kernels along the x-axis and y-axis respectively, is the convolution operator, s is the image signal;
[0174] Using the gradients of the x-axis and y-axis, the gradient magnitude of the image can be calculated as follows:
[0175]
[0176] Where P is the gradient amplitude;
[0177] Then the image is subjected to Gaussian low-pass filtering to obtain a smoothed image, which is calculated as follows:
[0178]
[0179] Among them, r g is the smoothed image, r is the original image, and g represents the Gaussian filter, which is defined as follows:
[0180]
[0181] Then, the saliency map is processed by combining the gradient amplitude of the original image and the gradient amplitude of the image smoothed by Gaussian low-pass filtering to obtain the saliency image corresponding to the original image. The saliency image is calculated as follows:
[0182]
[0183] Among them, M(r) is the saliency image, P r is the gradient amplitude of the original image, is the gradient amplitude of the smoothed image, C is a constant, Where L is the dynamic range of pixel values, and K is a scalar constant;
[0184] Saliency processing is to enhance the image by processing the high saliency value areas in the saliency image, so that these areas are more prominent in the combined image. The enhanced image can be obtained by enhancing the brightness, contrast, etc. of these areas of the original image. The calculation is as follows:
[0185] I enhanced =I original ·(1+k·M);
[0186] Among them, I enhanced is the enhanced image, I original is the original image, k is the preset enhancement coefficient, and M is the saliency map.
[0187] The local and global feature extraction network modules described in this embodiment are based on the Res2Net and SwinTransformer architectures. CNN is highly robust in extracting local context information, while Transformer is good at building global associations. The local and global feature extraction network modules specifically include:
[0188] Res2Net is an improved residual network architecture. It improves the network's expressiveness and performance by dividing the feature map in the residual block into multiple sub-feature maps and introducing multi-scale feature fusion between the sub-feature maps. Res2Net introduces features of multiple scales in a residual block, which enables the network to capture information of different scales at the same time. It fuses features of different scales in a hierarchical manner, improving the network's ability to capture complex scenes and details.
[0189] Assuming the input feature is X, Res2Net divides it into s scales, expressed as The feature of each scale is represented as X i , let Y i is the output feature of the i-th scale, then:
[0190] Y1=X1,Y i =Conv i (X i +Y i-1 ),;
[0191] Among them, Conv i represents the convolution function of the i-th scale, i = 2, 3, ..., s;
[0192] The final output feature is the concatenation of the features of each scale and then a residual connection is added, which is expressed as:
[0193]
[0194] Among them, Z is the final output feature, and X is the preset residual;
[0195] Swin Transformer is a transformer architecture for visual tasks. Unlike traditional convolutional neural networks (CNNs), Swin Transformer uses a self-attention mechanism to capture global and local information in images. Swin Transformer divides images into non-overlapping sub-patches and gradually increases the size of the patches in each layer to achieve multi-scale feature learning. Through windowing (i.e., performing self-attention calculations within local windows), Swin Transformer effectively reduces the amount of computation while maintaining the advantages of the self-attention mechanism. In order to capture cross-window context information, Swin Transformer introduces a sliding window mechanism so that information can be transferred between different windows. Assuming that the input feature is X, in order to capture cross-window context information, Swin Transformer introduces a sliding window mechanism, i.e., sliding a number of steps between different windows for attention calculations;
[0196] First, divide the feature X into windows:
[0197] X=Partition(X,window_size);
[0198] Then the self-attention calculation is performed on the features within each window, and the features within the window are linearly transformed:
[0199]
[0200] Y=Attention(Q,K,V)W o ;
[0201] Among them, Attention(Q,K,V) is the self-attention calculation function, Y is the linear transformation of the window feature, Q, K, V are query, key and value respectively, d k is the dimension of the key, W o is the linear transformation matrix;
[0202] Finally, the features within the window are merged into global features:
[0203] Z = Merge(Y);
[0204] The local and global feature extraction module (LGEM) combines convolutional neural networks (CNNs) and Transformers to extract local and global features, respectively. Its design concept is to use the local context information extraction capability of CNN and the global correlation construction capability of Transformer to maximize the use of information through interactive modeling. First, the initial features for input to the CNN stream and Transformer stream are extracted from two shallow blocks containing simple convolution operations to obtain and Then LGEM consists of a CNN stream and a Transformer stream composed of Res2Net and Swin Transformer, which extract and fuse features through interactive modeling to maximize the use of information. and The input Res2Net stream and SwinTransformer stream are processed separately, and CNN features and Transformer features are output respectively. The formula is as follows:
[0205]
[0206] where C(·) and T(·) represent the Res2Net101 and SwinTransformer encoder blocks, respectively. and They represent the CNN features and Transformer features output at stage s∈{1,2,3,…,n}, respectively, and Zs It represents the feature obtained by convolution and downsampling after s steps of multiplication of the distortion residual map and the distorted image. It provides the advantages of enhancing distortion perception information and preventing overfitting. The formula is:
[0207]
[0208] It is worth noting that in the Transformer flow, the distortion residual map is used to guide representation learning, which combines the self-attention mechanism to enhance the network's sensitivity and importance perception of the distortion residual area. The formula is as follows:
[0209] Q res = Flat(maxpool(Conv(E)));
[0210] Among them, E is the input distortion residual map, which is transformed by convolution operation, maximum pooling operation to reduce the feature scale, and finally expanded to obtain the query vector Q res , the Transformer’s self-attention is calculated as follows:
[0211]
[0212] Among them, key K x Sum value V x Derived from the distorted image, d k Represents the dimension of the key, query Q res It plays an important role in guiding the network to focus on the distorted regions, i.e., to concentrate on the distorted residual map.
[0213] More specific:
[0214] The local features are extracted by a network based on the Res2Net module, which includes an input module and a feature extraction module. The input module is used to preliminarily process the input image and convert it into a feature map suitable for further processing. The feature extraction module is used to perform multi-scale feature extraction on the feature map, which includes four stages:
[0215] The first stage is used to extract shallow features, including a convolution layer and multiple continuous multi-scale residual blocks. The convolution layer uses a convolution kernel size of 7×7 and a stride of 2. The total number of output feature maps is 64. The multi-scale residual block divides the input feature map into multiple sub-parts, each of which is convolved at a different scale, and then features are fused through cascading and 1x1 convolution kernels. Finally, the input is directly added to the fused output through residual connections.
[0216] The second stage is used to extract mid-level features, including a convolutional layer and multiple consecutive multi-scale residual blocks. The convolution kernel size of the convolutional layer is 3×3, the stride is 2, the total number of output feature maps is 128, and the structure of the multi-scale residual block is the same as that of the first stage;
[0217] The third stage is used to extract deep features, including a convolutional layer and multiple consecutive multi-scale residual blocks. The convolution kernel size of the convolutional layer is 3×3, the stride is 2, and the total number of output feature maps is 256. The structure of the multi-scale residual block is the same as the first two stages;
[0218] The fourth stage is used to further extract deep features, including a convolutional layer and multiple consecutive multi-scale residual blocks. The convolution kernel size of the convolutional layer is 3×3, the stride is 2, the total number of output feature maps is 512, and the structure of the multi-scale residual block is the same as the previous stage;
[0219] Global features are extracted by a network based on the SwinTransformer module, including a PatchPartition module and feature extraction module;
[0220] The Patch Partition module is a convolution operation with a convolution kernel size of 4×4, a convolution stride of 4, and a total of 48 feature maps. It performs multi-scale feature extraction on the image encoded into a feature vector after slicing, including the first stage, the second stage, the third stage, and the fourth stage;
[0221] The first stage is used to extract shallow features, including a Linear Embedding module and 2 consecutive Swin TransformerBlocks. The Linear Embedding module performs feature mapping for linear operations. The convolution kernel size is H / 4*W / 4, and the total number of feature maps is 96. The structure of the Swin Transformer Block connected in an even number is as follows: the input vector of the odd-numbered Swin Transformer Block is first normalized within the layer (LN regularization), and the standard multi-head self-attention module (W-MSA) is used to process the vector in the standard window. The processing result is added to the input and then normalized. The residual is connected to the multi-layer fully connected feedforward network and then output. The difference between the Swin Transformer Block with an even number and the Swin Transformer Block with an odd number is that the standard multi-head self-attention module (W-MSA) is replaced with a sliding window-based multi-head self-attention module (SW-MSA);
[0222] The second stage is used to extract mid-level features, including a Patch Merging module and two consecutive SwinTransformerBlocks. The Patch Merging module mainly reduces the size of the feature vector at the beginning of each stage to obtain hierarchical features.
[0223] The third stage includes a Patch Merging module and 18 consecutive SwinTransformer Blocks, and the fourth stage includes a PatchMerging module and 2 consecutive SwinTransformer Blocks. Both the third and fourth stages are used to extract deep features;
[0224] Among them, the calculation formula of the attention module is:
[0225]
[0226] Among them, Q is the query matrix corresponding to the input vector, K is the key matrix, and V is the value matrix. Their initial values are randomly generated. k is the dimension of the vector, and the calculation formula of the Softmax activation function is as follows:
[0227]
[0228] Among them, S j is the softmax value of the jth component in the vector, e j is the value of the jth component;
[0229] The calculation formula of multi-head attention (W-MSA) is as follows:
[0230] MultiHead(Q,K,V)=Concat(head1,…,head h )W1;
[0231] head i =Attention(Q i ,K i ,V i );
[0232] Among them, head i represents the i-th head, W1 represents the weight matrix;
[0233] The calculation of two consecutive SwinTransformerBlocks is as follows:
[0234]
[0235] in, and They represent the output features of the W-MSA layer and the MLP layer of the Swin Transformer Block module with odd numbers. W-MSA represents window-based multi-head attention, and SW-MSA represents moving window multi-head attention.
[0236] The details of the four-stage multi-scale features obtained by Swin Transformer feature extraction are as follows:
[0237] After passing through the first and second stage Patch Merging modules, the feature map size is (H, W are the height and width of the image, respectively, C is the number of channels through the first stage, which is 128, the same below). Then, after passing through the second and third stage Patch Merging modules, the feature map size obtained is Then, after passing through the Patch Merging modules in the third and fourth stages, the feature map size is Finally, the feature map size obtained after the fourth stage is
[0238] The channel and spatial attention module described in this embodiment is a lightweight attention mechanism module that integrates channel and spatial attention and can enhance the representation ability of convolutional neural networks. The channel and spatial attention module combines the channel attention mechanism (CAM) and the spatial attention mechanism (SAM). The goal of the channel attention module is to enhance feature representation through the importance weights of different channels. It assigns weights to each channel by capturing global spatial information. The specific steps are as follows:
[0239] First, the input feature map Perform global average pooling and global maximum pooling respectively to obtain two different feature descriptions:
[0240]
[0241] The above two feature descriptions are then input into a shared two-layer perceptron to generate two channel attention features:
[0242]
[0243] Among them, W0 and W1 are weight matrices, is a learnable weight matrix, σ is the sigmoid function, and r is the preset compression ratio coefficient;
[0244] Then the two feature descriptions are added element by element, and the channel attention weight is obtained through the sigmoid function:
[0245]
[0246] Then multiply the channel attention weight and the original feature map element by element to get the channel attention feature:
[0247] F channel =M c (F)⊙F;
[0248] The channel attention module is followed by the spatial attention module. The goal of the spatial attention module is to enhance feature representation by paying attention to the importance weights of different spatial positions. Unlike the channel attention module, the spatial attention module operates in the spatial dimension to highlight the salient features at different positions in the image. The specific steps are as follows:
[0249] First, the obtained channel attention features Perform pooling operations, perform global average pooling and global maximum pooling in the channel dimension, and obtain two feature images:
[0250]
[0251] The two feature images are then concatenated in the channel dimension to form a new concatenated feature image:
[0252]
[0253] The concatenated feature image is input into a 7×7 convolution layer, and the spatial attention weight coefficient is generated through the convolution operation:
[0254] M s (F channel )=σ(Conv(F concat ));
[0255] Among them, Conv represents the 7×7 convolution function, σ is the sigmoid function;
[0256] Finally, the feature image is multiplied element by element with the spatial attention weight coefficient to obtain the feature image after the channel and spatial attention module enhancement representation. The feature image after module enhancement is then output as the module output feature image:
[0257] F spatial =M s (F channel )⊙F channel ;
[0258] Fout =F spatial ;
[0259] Among them, F out is the module output feature image, F spatial is the feature image after module enhancement, and ⊙ is the Hadamard product.
[0260] After the channel and spatial attention modules obtain the final output feature image, they will be regressed through two fully connected layers (FC) to obtain an objective score, and then a five-parameter nonlinear fitting function is used to map the objective score to the subjective score, as follows:
[0261] F fc1 =ReLU(W1F out +b1);
[0262] Score objective =W2F fc1 +b2;
[0263] Among them, F fc1 is the output of the first fully connected layer, and is the weight matrix and bias vector of the first fully connected layer, D is the output dimension of the first fully connected layer, and It is the weight matrix and bias vector of the second fully connected layer, and finally outputs a scalar as the objectivity score of the second fully connected layer of the image;
[0264] In the image quality assessment (IQA) task, the five-parameter nonlinear fitting function (5PL model) can be used to map the objective quality score to the subjective score to better conform to the nonlinear relationship of human subjective perception. The objective score is mapped according to the preset parameter nonlinear fitting model, and the formula for obtaining the subjective score is as follows:
[0265]
[0266] Among them, Score subjective is the subjective score, η i are preset fitting parameters, i=1,2,3,4,5.
[0267] S3. Preprocessing the training image set in the image data set and inputting it into the full-reference image quality initial evaluation model for training to obtain a trained full-reference image quality evaluation model;
[0268] S4. Inputting the real distorted image set to be evaluated into the trained full-reference image quality evaluation model for processing to obtain an image quality score;
[0269] Specifically, the resolution of the images in the training image set is adjusted, specifically, they are first scaled to 512×384, and then randomly cut to multiple image slices of size 384×384 to meet the input specifications of Res2Net and SwinTransformer and retain more information of the original image that is useful for quality evaluation. In this embodiment of the scheme, the number of random cuts of the same image is set to 5 to meet the large data volume requirements of Res2Net and SwinTransformer. The image slices are input into the trained full-reference image quality evaluation model for processing to obtain the corresponding image prediction score value. In this embodiment of the scheme, the final score of an image predicted by the evaluation model will be obtained by averaging the prediction scores of 5 representing an image. The image quality score of an image predicted by the model is:
[0270]
[0271] Among them, Score is the image quality score, n is the number of image slices, Pred j is the image prediction score corresponding to the j-th image slice;
[0272] In the model training process of this embodiment of the scheme, the mean square error is used as the loss function, and the mean square error is defined as follows:
[0273]
[0274] Among them, B represents the size of a batch during training, which is set to 8 in the experiment, x i Represents the prediction score of the model, y i Represents the proxy score of the image slice, which is replaced by the label score of the original image.
[0275] It should be noted that two indicators were used in the process of evaluating the experimental results: Pearson linear correlation coefficient (PLCC) and Spearman rank correlation coefficient (SROCC);
[0276] Among them, PLCC mainly measures the accuracy of the prediction of the objective quality evaluation method, and its expression is:
[0277]
[0278] Where N is the number of images, s i is the subjective score of the i-th image among N images, is the mean of the subjective scores, p i is the objectivity score of the i-th image among N images, is the mean of the objective scores;
[0279] SROCC is an indicator to measure the monotonic consistency of objective quality prediction scores. Its value is only related to the ranking of the image in the prediction results, ignoring the relative distance between the predicted value and the true value. Its expression is:
[0280]
[0281] Among them, d i is the difference between the ranking numbers of image i in the subjective score and the objective score, and N is the number of images;
[0282] The closer PLCC and SROCC are to 1, the better the proposed method is.
[0283] The second aspect of the present invention also discloses a real distortion full-reference image quality assessment system, including a memory and a processor, wherein the memory includes a real distortion full-reference image quality assessment method program, and when the real distortion full-reference image quality assessment method program is executed by the processor, the steps of the real distortion full-reference image quality assessment method as described above are implemented.
[0284] A third aspect of the present invention provides a computer-readable storage medium, which includes a real-distortion full-reference image quality assessment method program. When the real-distortion full-reference image quality assessment method program is executed by a processor, the steps of the real-distortion full-reference image quality assessment method as described in any one of the above items are implemented.
[0285] The present invention discloses a method, system and medium for evaluating the quality of a real distorted full-reference image. The method extracts local and global features through interactive modeling of a backbone network using Res2net and SwinTransformer as feature extraction through deep learning technology. The saliency map detection module is used to guide the model to focus on the area of human visual attention. The distortion-aware reconstruction network is used to generate a corresponding distortion residual map to guide attention calculation and perform residual connection to improve the performance of the network. The channel and spatial attention modules are connected to enhance the representation of features, and weights are allocated and learned. The distortion residual map obtained by the distortion-aware reconstruction network is used to guide network learning. Then, two fully connected layers predict scores based on the quality perception features learned by the dual attention mechanism. Finally, the scores are mapped to subjective scores through a five-parameter nonlinear fitting function. The image is evaluated comprehensively and objectively, so that the model can make an evaluation of the image quality that is consistent with the subjective perception of the human eye.
[0286] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.
[0287] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0288] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.
[0289] Those skilled in the art can understand that all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions, and the aforementioned program can be stored in a readable storage medium. When the program is executed, it executes the steps of the above method embodiments; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), disks or optical disks, and other media that can store program codes.
[0290] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.
Claims
1. A method for evaluating the quality of a true-distortion full-reference image, characterized in that: The following steps are involved: Obtaining a full reference image quality assessment dataset including a training image set with real distortion, and generating an image data set; Obtain the initial full-reference image quality assessment model, including the distortion-aware reconstruction network module, the saliency map detection module, the local and global feature extraction network modules, and the channel and spatial attention modules; Preprocessing the training image set in the image data set and inputting the training image set into the full-reference image quality initial evaluation model for training to obtain a trained full-reference image quality evaluation model; The real distorted image set to be evaluated is input into the trained full-reference image quality evaluation model for processing to obtain an image quality score.
2. The method for evaluating true distortion full-reference image quality according to claim 1, characterized in that: The distortion-aware reconstruction network module comprises: The distortion-aware reconstruction network module includes an encoder structure, a decoder structure and a Short-cut layer; The encoder structure includes a plurality of sub-layers, including a plurality of convolutional layers and activation functions, and extracts features of the distorted image according to the encoder structure; The calculation formula for extracting distorted image features by the encoder structure is: F=ReLU(Conv(I d )); Among them, F is the degradation graph feature extracted by the encoder structure, I d is the degradation graph, Conv(·) is the convolutional layer, and ReLU(·) is the activation function; The decoder structure comprises a plurality of sub-layers, including a plurality of deconvolution layers and activation functions, and extracts degradation map features according to the encoder structure and reconstructs and generates a corresponding distortion residual map; The calculation formula of the distortion residual map reconstructed by the decoder structure is: E=ReLU(ConvTranspose2d(F)); Among them, E is the distortion residual map reconstructed by the decoder structure, F is the degradation map feature extracted by the encoder structure, ConvTranspose2d(·) is the deconvolution layer, and ReLU(·) is the activation function; The Short-cut layer obtains a distortion residual map according to the reference image and the distorted image, and then obtains a picture by comparing and splicing the distortion residual map and the degradation map, and performs difference comparison with the reference image to train the distortion-aware reconstruction network module; The distortion-aware reconstruction network module also includes an error-aware reconstruction network model, which includes a mean absolute error and a structural similarity index; The function calculation formula of the mean absolute error is: in, is the loss function of mean absolute error, |x i -y i | is the absolute value of the difference between the i-th corresponding pixels of the two images; The functional calculation formula of the structural similarity index is: in, is the loss function of the structural similarity index, μ x and μ y are the average values of the contrast features of the two images x and y, σ x and σ y are the standard deviations of the contrast features of the two images x and y, σ xy is the contrast feature covariance of x and y, c1 and c2 are the preset feature constant coefficients; Performing objective function minimization processing on the mean absolute error and the structural similarity index according to the error-aware reconstruction network model; The calculation formula for minimizing the objective function of the error-aware reconstruction network is: Among them, I d is the degradation graph, E is the distortion residual graph reconstructed by the decoder structure, and I ref is the corresponding reference graph, and λ is the preset weighting coefficient.
3. The method for evaluating the quality of a true-distortion full-reference image according to claim 2, characterized in that: The saliency map detection module comprises: The saliency map detection module calculates the gradient of the image in the x-axis and y-axis directions through convolution operation, and the calculation formula is: Among them, P x and P y are the gradients of the image in the x-axis and y-axis directions, h x and h y are the convolution kernels along the x-axis and y-axis respectively, is the convolution operator, s is the image signal; The gradient amplitude of the image is calculated according to the gradients in the x-axis and y-axis directions, and the calculation formula is: Where P is the gradient amplitude; Then perform Gaussian low-pass filtering on the image to obtain a smoothed image. The calculation formula is: Among them, r g is the smoothed image, r is the original image, and g represents the Gaussian filter; The function formula of Gaussian filter is: Then, the saliency map is processed by combining the gradient amplitude of the original image and the gradient amplitude of the image smoothed by Gaussian low-pass filtering to obtain a saliency image corresponding to the original image; The processing calculation formula of the saliency image is: Among them, M(r) is the saliency image, P r is the gradient amplitude of the original image, is the gradient amplitude of the smoothed image, C is a constant, Where L is the dynamic range of pixel values, and K is a scalar constant; Performing enhancement processing on the original image according to the saliency image to obtain an enhanced image; The processing calculation formula of the enhanced image is: I enhanced =I original ·(1+k·M); Among them, I enhanced is the enhanced image, I original is the original image, k is the preset enhancement coefficient, and M is the saliency map.
4. The method for evaluating true distortion full-reference image quality according to claim 3, characterized in that: The local and global feature extraction network module comprises: The local and global feature extraction network modules include Res2Net and SwinTransformer: According to Res2Net, feature X is divided into s scales, expressed as The features of each scale are represented by X i ; Y i is the output feature of the i-th scale in the s-th scale, Y1=X1,Y i =Conv i (X i +Y i-1 ); Among them, Conv i is the convolution function of the i-th scale, i = 2, 3, ...s; Perform splicing processing according to the features of each size and combine them with preset residuals to obtain final output features; The calculation formula of the final output feature is: Among them, Z is the final output feature, and X is the preset residual; Perform attention calculation on the feature X according to Swin Transformer, including: Divide the feature X into windows: X=Partition(X,window_size); Perform self-attention calculation and linear transformation on the features in each divided window: Y=Attention(Q,K,V)W o ; Among them, Attention(Q,K,V) is the self-attention calculation function, Y is the linear transformation of the window feature, Q, K, V are query, key and value respectively, d k is the dimension of the key, W o is the linear transformation matrix; Merge the features in each window to obtain the global features: Z = Merge(Y); Among them, Z is the global feature, Merge(·) is the merging function; The initial features are extracted by two shallow convolution blocks, and we get and And input into the Res2Net stream and SwinTransformer stream for processing respectively, and output CNN features and Transformer features respectively; The output formula of the CNN feature and Transformer feature is: Among them, C(·) and T(·) are the encoder blocks of Res2Net and Swin Transformer respectively. and are the CNN features and Transformer features output at stage s∈{1,2,3,…,n}, respectively, Z s It is the feature obtained by multiplying the distorted residual map and the distorted image through s steps of convolution and downsampling.
5. The method for evaluating the quality of a true-distortion full-reference image according to claim 4, characterized in that: The channel and spatial attention module comprises: The channel and spatial attention module includes a channel attention module CAM and a spatial attention module SAM; Get the feature image F, And perform global average pooling and global maximum pooling respectively to obtain the corresponding feature descriptions: Describe the characteristics Input the preset shared two-layer perceptron for processing respectively, and obtain the corresponding channel attention features respectively: Among them, W0 and W1 are weight matrices, σ is the sigmoid function, r is the preset compression ratio coefficient; The channel attention feature Through the sigmoid function, the channel attention weight coefficient is obtained: The channel attention weight coefficient M c (F) Processing with the feature image to obtain the weighted channel attention feature: F channel =M c (F)⊙F; For the weight channel attention feature F channel , Perform global average pooling and global maximum pooling respectively to obtain the corresponding feature images: According to the feature image Perform stitching processing to obtain the stitching feature image: The spliced feature image F concat The spatial attention weight coefficient is obtained by processing with the sigmoid function: M s (F channel )=σ(Conv(F concat )); Among them, Conv is the convolution function, σ is the sigmoid function; The spatial attention weight coefficient M s (F channel ) is processed with the feature image to obtain the feature image after module enhancement processing: F spatial =M s (F channel )⊙F channel ; Outputting the feature image after the module enhancement processing as a module output feature image; F out =F spatial ; Among them, F out Output feature image for the module, F spatial is the feature image after module enhancement, and ⊙ is the Hadamard product.
6. The method for evaluating the quality of a true-distortion full-reference image according to claim 5, characterized in that: The channel and spatial attention module also includes: According to the output feature image of the module, regression processing is performed through a preset fully connected layer to obtain an objective score; F fc1 =ReLU(W1F out +b1); Score objective =W2F fc1 +b2; Among them, F fc1 is the output of the first fully connected layer, W1 is the weight matrix of the first fully connected layer, b1 is the preset bias vector of the first fully connected layer, ReLU(·) is the activation function, Score objective is the objective score of the second fully connected layer, W2 is the weight matrix of the second fully connected layer, b2 is the preset bias vector of the second fully connected layer, Mapping the objective scores according to a nonlinear fitting model with preset parameters to obtain subjective scores; Among them, Score subjective is the subjective score, η i are preset fitting parameters, i=1,2,3,4,5.
7. The method for evaluating true distortion full-reference image quality according to claim 6, characterized in that: The step of inputting the real distorted image set to be evaluated into the trained full-reference image quality evaluation model for processing to obtain an image quality score comprises: Acquire multiple image slices in a set of real distorted images to be evaluated; Inputting the image slice into the trained full-reference image quality assessment model for processing to obtain a corresponding image prediction score value; Processing the image slices according to the image prediction score values corresponding to the plurality of image slices to obtain an image quality score; The calculation formula of the image quality score is: Among them, Score is the image quality score, n is the number of image slices, Pred j is the image prediction score corresponding to the j-th image slice.
8. A true distortion full reference image quality assessment system, characterized in that: The system includes: a memory and a processor, wherein the memory includes a program of a real distortion full reference image quality assessment method, and when the program of the real distortion full reference image quality assessment method is executed by the processor, the following steps are implemented: Obtaining a full reference image quality assessment dataset including a training image set with real distortion, and generating an image data set; Obtain the initial full-reference image quality assessment model, including the distortion-aware reconstruction network module, the saliency map detection module, the local and global feature extraction network modules, and the channel and spatial attention modules; Preprocessing the training image set in the image data set and inputting the training image set into the full-reference image quality initial evaluation model for training to obtain a trained full-reference image quality evaluation model; The real distorted image set to be evaluated is input into the trained full-reference image quality evaluation model for processing to obtain an image quality score.
9. The real distortion full reference image quality assessment system according to claim 8, characterized in that: The distortion-aware reconstruction network module comprises: The distortion-aware reconstruction network module includes an encoder structure, a decoder structure and a Short-cut layer; The encoder structure includes a plurality of sub-layers, including a plurality of convolutional layers and activation functions, and extracts features of the distorted image according to the encoder structure; The calculation formula for extracting distorted image features by the encoder structure is: F=ReLU(Conv(I d )); Among them, F is the degradation graph feature extracted by the encoder structure, I d is the degradation graph, Conv(·) is the convolutional layer, and ReLU(·) is the activation function; The decoder structure comprises a plurality of sub-layers, including a plurality of deconvolution layers and activation functions, and extracts degradation map features according to the encoder structure and reconstructs and generates a corresponding distortion residual map; The calculation formula of the distortion residual map reconstructed by the decoder structure is: E=ReLU(ConvTranspose2d(F)); Among them, E is the distortion residual map reconstructed by the decoder structure, F is the degradation map feature extracted by the encoder structure, ConvTranspose2d(·) is the deconvolution layer, and ReLU(·) is the activation function; The Short-cut layer obtains a distortion residual map according to the reference image and the distorted image, and then obtains a picture by comparing and splicing the distortion residual map and the degradation map, and performs difference comparison with the reference image to train the distortion-aware reconstruction network module; The distortion-aware reconstruction network module also includes an error-aware reconstruction network model, which includes a mean absolute error and a structural similarity index; The function calculation formula of the mean absolute error is: in, is the loss function of mean absolute error, |x i -y i | is the absolute value of the difference between the i-th corresponding pixels of the two images; The functional calculation formula of the structural similarity index is: in, is the loss function of the structural similarity index, μ x and μ y are the average values of the contrast features of the two images x and y, σ x and σ y are the standard deviations of the contrast features of the two images x and y, σ xy is the contrast feature covariance of x and y, c1 and c2 are the preset feature constant coefficients; Performing objective function minimization processing on the mean absolute error and the structural similarity index according to the error-aware reconstruction network model; The calculation formula for minimizing the objective function of the error-aware reconstruction network is: Among them, I d is the degradation graph, E is the distortion residual graph reconstructed by the decoder structure, and I ref is the corresponding reference graph, and λ is the preset weighting coefficient.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a real-distortion full-reference image quality assessment method program. When the real-distortion full-reference image quality assessment method program is executed by a processor, the steps of a real-distortion full-reference image quality assessment method as described in any one of claims 1 to 7 are implemented.
Citation Information
Cited By
Depth map dynamic restoration method integrating quality perception
CN119693279A
Medical image quality rapid detection method
CN120125570A
A rapid detection method for medical image quality
CN120125570B
Image semantic reconstruction quality evaluation method and device based on attention weighting
CN120580214A
