Multi-stage pre-training based no-reference image quality detection method and system

Abstract

This invention belongs to the field of image processing, specifically relating to a method and system for no-reference image quality detection based on multi-stage pre-training. The method includes sequentially extracting features from the image to be detected using a first feature extractor and a second feature extractor to obtain image features. The first feature extractor is an encoder in an image restoration network, and the second feature extractor consists of multiple parallel, structurally consistent feature extraction sub-networks trained as a whole on different IQA datasets using image quality assessment methods. Based on the image features, a second detection result is obtained through an image quality detection network. The image quality detection network is constructed based on cross-dataset quality detection correlation and a self-attention mechanism. This invention improves the accuracy and generalization of quality detection for different categories of images.

Description

Technical Field

[0001] This invention belongs to the field of image processing, specifically relating to a method and system for no-reference image quality detection based on multi-stage pre-training. Background Technology

[0002] Convolutional Neural Networks (CNNs) possess excellent feature extraction capabilities, making them a common implementation method for Image Quality Assessment (IQA) models in recent years. Early implementations typically used shallow networks, resulting in mediocre performance. With the advent of deep networks such as VGG and ResNet, IQA models often employ these networks as feature extractors, achieving superior results compared to shallow networks. However, limited by the size of IQA datasets, IQA models using deep networks are prone to overfitting and exhibit poor generalization ability.

[0003] Currently, there are many open-source IQA datasets available, such as CSIQ, LIVE, and TID2013. The simplest way to solve the above problem is to directly train multiple IQA datasets together. However, different IQA datasets have different annotation formats and score ranges. For example, CSIQ uses DMOS in the range of [0,1], TID2013 uses MOS in the range of [0,10], and LIVE uses DMOS in the range of [0,100]. Simple normalization can lead to significant subjective visual quality differences between two images with the same score.

[0004] To address the lack of labeled IQA data, many methods propose pre-training on other tasks or datasets and then fine-tuning the network on the target IQA dataset. For example, pre-training can be done on the ImageNet dataset for image classification tasks, image quality ranking tasks, or image classification and distortion type tasks.

[0005] However, image classification pre-training tasks focus more on extracting semantic information from images and pay less attention to distortion information, which will affect the subsequent IQA model's judgment of image quality. In addition, although image quality ranking pre-training tasks are closely related to IQA tasks, the former can only rank two distorted images from the same reference image that contain the same type of distortion, which limits the network's ability to learn quality assessments for different image content and different distortion types. Summary of the Invention

[0006] To address the aforementioned problems in existing technologies, namely, the poor accuracy and weak generalization of quality detection for different image categories, this invention proposes a no-reference image quality detection method based on multi-stage pre-training, characterized by the following steps:

[0007] The first feature extractor and the second feature extractor sequentially extract features from the image to be detected to obtain image features. The first feature extractor is the encoder in the image restoration network, and the second feature extractor is a series of parallel feature extraction sub-networks with the same structure, which are trained as a whole based on different IQA datasets using image quality assessment methods.

[0008] Based on the image features, the detection results are obtained through an image quality detection network; the image quality detection network is constructed based on cross-dataset quality detection correlation and self-attention mechanism.

[0009] In some preferred embodiments, the second feature extractor is constructed as follows:

[0010] Multiple image quality assessment networks with the same network structure are set up in parallel. After being trained on a corresponding number of IQA datasets, the feature extraction part is selected as the second feature extractor.

[0011] In some preferred embodiments, a two-stage weight update method is used to train the image quality assessment network:

[0012] Each image quality assessment network is trained at level one based on the absolute difference between the output evaluation value and the sample labeled evaluation value as the loss function.

[0013] Secondary training is performed on multiple image quality assessment networks that are combined in parallel, based on the sum of the loss functions of all image quality assessment networks.

[0014] In some preferred embodiments, the cross-dataset quality detection correlation calculation includes: calculating f1, f2...f N The correlation between pairs of features, and the two sets of features obtained after feature aggregation and feature separation based on the correlation; where f1, f2...f N These are the features extracted from the N sub-networks of the second feature extractor.

[0015] In some preferred embodiments, f1, f2...f N Methods for calculating pairwise correlation include:

[0016] N 1×1 convolutional layers are used to process f1, f2...f N Calculate the key vectors k1, k2, ..., k N ;

[0017] For k1, k2...k N The correlation between each pair is calculated to obtain c. ij ;

[0018] in,

[0019]

[0020] c ij For feature f i and f j Correlation between them; symbols This indicates a point-to-point multiplication operation.

[0021] In some preferred embodiments, the methods for feature aggregation and feature separation are as follows:

[0022] f ij =f i +c ij f j (i≠j)

[0023]

[0024] Among them, f ij f i and f j Features after feature aggregation f i and f j Features after feature separation.

[0025] In some preferred embodiments, the image quality detection network employs a self-attention mechanism, based on f ij , Calculate the globally strongly correlated feature y and the globally weakly correlated feature.

[0026] In another aspect, the present invention proposes a no-reference image quality detection system based on multi-stage pre-training, comprising a first unit and a second unit;

[0027] The first unit includes a first feature extractor and a second feature extractor connected in sequence, used to extract image features from the input image; the first feature extractor is an encoder in the image restoration network; the second feature extractor is a set of multiple parallel feature extraction sub-networks with consistent structure, which are obtained after overall training using image quality assessment methods based on different IQA datasets;

[0028] The second unit includes an image quality detection network; the image quality detection network is constructed based on cross-dataset quality detection correlation and self-attention mechanism, and obtains image quality detection results based on the image features output by the second feature extractor.

[0029] A third aspect of the present invention provides an electronic device comprising: at least one processor; and a memory communicatively connected to at least one of the processors; wherein the memory stores instructions executable by the processor for implementing the above-described multi-stage pre-training-based no-reference image quality detection method.

[0030] In a fourth aspect, the present invention provides a computer-readable storage medium storing computer instructions for execution by the computer to implement the above-described multi-stage pre-training-based no-reference image quality detection method.

[0031] The beneficial effects of this invention are:

[0032] (1) In this invention, the encoder in the image restoration network is used as the first feature extractor, which is obtained through training the image restoration network. On the one hand, this is because image restoration and IQA tasks have many similarities. Both take distorted images as input, and both need to perceive the semantics and distortion information of the image to complete the restoration task and the image quality score prediction task. On the other hand, the training of the image restoration task does not require labeled data, which is very easy to obtain. It is only necessary to apply various distortion factors to the high-definition image to generate distorted images, thereby obtaining training image pairs. Therefore, generating a large number of distorted-high-definition image pairs and pre-training them on the image restoration task can help the IQA model fully learn prior knowledge about image semantics and distortion, which helps to improve the accuracy of the image quality detection results of this invention.

[0033] (2) This invention is based on multiple image quality assessment networks with the same network structure set in parallel. The network is trained as a whole on the IQA dataset with the corresponding number of categories. Each image quality assessment network learns the image quality detection rules of one IQA dataset. The feature extraction part is retained as the second feature extractor. Based on the cross-dataset quality detection correlation network structure part in the subsequent image quality detection network, the strong correlation features and weak correlation features of each sub-network are learned. In this way, the commonalities and differences of the scoring rules of these IQA datasets are found. The learned image quality detection rules are used as prior knowledge to further improve the accuracy and generalization ability of the image quality detection results of this invention. Attached Figure Description

[0034] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0035] Figure 1 This is a schematic diagram of the general process of the IQA model;

[0036] Figure 2This is a flowchart of a no-reference image quality assessment method that uses the restored image as a reference image;

[0037] Figure 3 These are examples of composite distorted images where the degree of distortion gradually increases from left to right;

[0038] Figure 4 This is a flowchart of an image quality assessment learning method based on image quality ranking pre-training.

[0039] Figure 5 This is a schematic diagram of the process of a no-reference image quality detection method based on multi-stage pre-training according to an embodiment of the present invention;

[0040] Figure 6 This is a schematic diagram of the selection of various network structures and the image quality detection logic in the method of this invention. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions in the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this invention, not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0042] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0043] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other.

[0044] No-reference image quality assessment involves predicting image quality scores in the absence of a reference image. For example... Figure 1 As shown, an IQA model generally consists of two parts: a feature extractor (such as a CNN) and a regressor (such as an SVM or neural network). The former is used to extract features from distorted images, and the latter is used to map the features to image quality scores.

[0045] Compared to image quality assessment with references, image quality assessment without references lacks any comparative information, making prediction more difficult and resulting in lower accuracy. To address this, many methods introduce image restoration models, such as... Figure 2As shown, the distorted image is first restored to a high-quality image with the distortion removed, and then this restored image is used as a reference image and input together with the original distorted image into the IQA model. However, this method is very sensitive to the quality of the restored image; if the restoration is poor, it will lead to inaccurate score predictions.

[0046] IQA models are typically trained using supervised learning methods. This involves calculating a loss function between the image quality scores predicted by the IQA model and the manually labeled image quality scores, and then using this loss function to update the weights of the IQA model. However, due to the high cost and time-consuming nature of image quality score labeling, existing IQA datasets (such as CSIQ and LIVE) are relatively small, containing only a few hundred training images. This makes it difficult to effectively train deep neural networks, resulting in limited accuracy and generalization of IQA models.

[0047] To address the issue of insufficient labeled data, some methods propose pre-training by ranking image quality as initialization parameters for the IQA model. First, a set of high-quality images (such as the Waterloo dataset containing 4744 high-resolution images) is acquired. Distortion factors (such as noise, blur, overexposure, underexposure, color deviation, low contrast, etc.) are artificially added to these high-quality images, and the degree of distortion is divided into five levels from least to most severe. Figure 3 As shown (the first row is a blurred and distorted image, and the second row is a noisy and distorted image), for each high-quality image, a series of distorted images with varying degrees of distortion from weak to strong can be generated. Two images can be randomly selected from these distorted images to form an image pair, and their quality ranking can be used as labels.

[0048] Then, as Figure 4 As shown, a Siamese network is constructed. A pair of images, a and b, is selected from the generated distorted image pairs and input into two separate networks. The networks output predicted quality scores Sa and Sb for the two images, and the values ​​of Sa and Sb are compared. If the order of Sa and Sb matches the labeled quality order, the loss function is 0; otherwise, the loss function is 1. This loss function is used to pre-train the Siamese network. After pre-training, one network is selected from the Siamese network and fine-tuned on the target IQA dataset.

[0049] Another classic approach, as proposed in the literature, involves pre-training two CNNs on image classification and distortion classification tasks, respectively. S-CNN and VGG-16 employ the same network architecture, learning image semantics and distortion-related knowledge through image classification and distortion classification pre-training tasks, respectively. After pre-training, the entire network is fine-tuned on the target IQA dataset. S-CNN is responsible for extracting distortion features from the input image, while VGG-16 is responsible for extracting semantic features. Finally, bilinear pooling is used to fuse the semantic and distortion features, which are then input into the fully connected (fc) layer to predict the image quality score.

[0050] IQA dataset: Each sample in the dataset contains a distorted image and a human-annotated image quality score. The IQA dataset is divided into the naturally distorted IQA dataset and the synthetically distorted IQA dataset. The human-annotated image quality scores are mainly in two forms: MOS (Mean Opinion Score) and DMOS (Differential Opinion Score).

[0051] The goal of this invention is to address the problem of insufficient labeled data for IQA (Image Quality Assurance) by leveraging pre-training techniques and external data. Image semantics, distortion, and human scoring rules are crucial knowledge that IQA models need to master. To address this, this invention designs a multi-stage pre-training task to acquire this prior knowledge. This prior knowledge will serve as the initial weights for the IQA model, enabling it to achieve good prediction accuracy and generalization even when trained on small-scale labeled datasets.

[0052] This invention provides a no-reference image quality detection method based on multi-stage pre-training, such as... Figure 5 As shown, it includes the following steps:

[0053] The first feature extractor and the second feature extractor sequentially extract features from the image to be detected to obtain image features. The first feature extractor is the encoder in the image restoration network, and the second feature extractor is a series of parallel feature extraction sub-networks with the same structure, which are trained as a whole based on different IQA datasets using image quality assessment methods.

[0054] Based on the image features, the detection results are obtained through an image quality detection network; the image quality detection network is constructed based on cross-dataset quality detection correlation and self-attention mechanism.

[0055] In the second feature extractor, each feature extraction sub-network corresponds to an image quality assessment network that learns image quality detection rules for an IQA dataset. To further improve the accuracy of the detection results, the image quality detection network can be trained and optimized using a small dataset corresponding to the images of the detection category before use.

[0056] To more clearly illustrate the multi-stage pre-training-based no-reference image quality detection method of this invention, as follows: Figure 6 As shown below, the construction and training of the first feature extractor, the second feature extractor, and the image quality detection network will be described in general.

[0057] 1. First Feature Extractor

[0058] An image restoration network is constructed, and an image restoration dataset is prepared to train the constructed network. The decoder in the trained network structure is selected as the first feature extractor.

[0059] A method for obtaining a trained image restoration network based on image restoration task training:

[0060] Step 101: Prepare the image restoration dataset. Collect a large number of high-resolution images as reference images, add different types and intensities of distortion factors to the high-resolution images to generate distorted images, and form an image pair with each distorted image and its corresponding high-resolution image;

[0061] Step 102: Construct an image restoration network. For example, the UNet network contains an encoder and a decoder. The encoder extracts features from the input image, and the decoder reconstructs the image from these features.

[0062] Step 103: Train the image restoration network. Take any image pair from the image pairs generated in Step 101, containing a distorted image I and its corresponding high-resolution image. The distorted image I is input into the encoder of the image restoration network constructed in step 102. The encoder outputs a feature map, which is then input into the decoder to output a restored image. Calculate the restored image and high-definition images The loss function is used to update the weights of the image restoration network.

[0063] The encoder is selected from the trained image restoration network as the first feature extractor.

[0064] 2. Second Feature Extractor

[0065] Collect several IQA datasets (assuming there are N), and build N image quality assessment networks in parallel based on the same network architecture. Each network corresponds to one IQA dataset. After training, retain the feature extraction part as a second feature extractor.

[0066] The image quality assessment network with parallel settings is trained using a multi-branch, multi-dataset joint training method:

[0067] Step 201: Collect several IQA datasets (assuming there are N datasets, D1, D2...D...). N For example, currently open-source datasets such as CSIQ, LIVE, TID2013, and Kadid will normalize all image quality scores labeled in these datasets into a unified labeling format and score range;

[0068] Step 202: Construct multiple subnetworks with the same structure (h1, h2...h... N First, the encoder of the trained image restoration network is taken as the first feature extractor. Then, N sub-network branches (image quality assessment network) are connected after the first feature extractor. Each sub-network is specifically responsible for learning the mapping from image features to image quality scores on an IQA dataset.

[0069] Step 203: Train the sub-network by randomly selecting one sample from each of the N IQA datasets collected in Step 201. Each sample contains a distorted image I. n and an image quality score s n , n (1≤n≤N) represents the index of N IQA datasets, and the distorted image I n Input the first feature extractor, output the first image feature, then input the first image feature into the nth sub-network, and output the predicted image quality score. Calculate the loss function L of the nth subnetwork n The loss function is used to train and optimize the nth sub-network, and the weight parameters in the network are updated (first-level training). At the same time, the sum of the loss functions of all sub-networks is used to optimize the combination of N sub-networks to achieve fine-tuning of the weights (second-level training).

[0070] After the N sub-networks set up in parallel are trained, their feature extraction parts are retained to form a second feature extractor.

[0071] 3. Image quality detection network

[0072] The image quality detection network is built based on cross-dataset quality detection correlation and self-attention mechanism. It is trained and optimized using a small-scale target IQA dataset, based on a first feature extractor, a second feature extractor, and an image quality detection network connected in sequence.

[0073] Optimize the image quality detection network using a fine-tuning method based on the target dataset:

[0074] Step 301: Construct the target IQA dataset, collect distorted images, and have scorers score each distorted image. Finally, take the average of all scores for each image as the labeled score for that image, and normalize the labeled score to the same labeled format and range as in Step 201.

[0075] Step 302: Construct an image quality detection network. Take the encoder of the trained image restoration network as the first feature extractor and the feature extraction part of the trained sub-network as the second feature extractor. Then connect an image quality detection network after the second feature extractor. This network is used to learn the mapping from image features to image quality scores (image quality detection results) on the target IQA dataset constructed in step 301.

[0076] Step 303: The image quality detection network described in step 302 includes a cross-dataset quality assessment correlation module and a self-attention module, used to mine the correlation of human scoring rules learned by the second feature extractor on multiple IQA datasets. Assume that features f1, f2...f are extracted from N networks of the second feature extractor respectively. N In the cross-dataset quality assessment relevance module, N 1×1 convolutional layers are first used to evaluate f1, f2, ..., f... N Calculate the key vectors k1, k2, ..., k N Then, using equation (1) to evaluate k1, k2...k N Calculate the correlation between each pair of pairs:

[0077]

[0078] Among them, symbols This indicates a point-to-point multiplication operation; c ij The larger the value, the greater the characteristic f. i and f j The greater the correlation between them;

[0079] Then, f is processed using equation (2). i and f j Aggregate features with strong correlation:

[0080] f ij =f i +c ij f j ,(i≠j) (2)

[0081] And through equation (3) for f i and f j Separate features with weak correlation:

[0082]

[0083] f1, f2...f N After calculating the correlations pairwise and performing feature aggregation and feature separation, we obtain f. 12 f 23 ...f ij (i≠j), and Then, the two sets of features are input into the self-attention module, with the first set of features (f) 12 f 23 ...f ij Taking the input to the self-attention module as an example, the feature map x is first concatenated along the channel, and then passed through three 1×1 convolutional layers in parallel to obtain three feature maps δ(x), φ(x), and τ(x). Then, the three feature maps are calculated using equation (4) to obtain the globally strongly correlated feature y:

[0084]

[0085] Where Γ(x) represents the number of pixels in feature maps δ(x), φ(x), and τ(x), and T represents the transpose operation. When the self-attention module inputs the second set of features, it performs the same operation and outputs globally weakly correlated features. Finally, y and The images are stitched together along the channel layers and then passed through several convolutional layers to output the predicted image quality score (image quality detection result).

[0086] Step 304: Train the image quality detection network. Randomly select a sample from the target IQA dataset constructed in Step 301. Each sample contains a distorted image I and an image quality score s. The distorted image I is sequentially processed by the first and second feature extractors to output image features. Then, these image features are input into the image quality detection network to output the predicted image quality score. The loss function is calculated and used to optimize the image quality detection network, update the network weights, and fine-tune the weights after optimizing the first and second feature extractors.

[0087] An image quality detection system is constructed based on a first feature extractor, a second feature extractor, and an image quality detection network configured in a sequential manner. This system performs image quality detection on the input image and obtains the detection results. The accuracy of the detection results for similar images on the target IQA dataset will be further improved.

[0088] To further illustrate the method of the present invention in detail, the construction and training of the first feature extractor, the second feature extractor, and the image quality detection network are described in detail below with reference to specific embodiments.

[0089] Step A101: Use the Waterloo dataset to create an image restoration dataset. The Waterloo dataset contains 4744 high-resolution images as reference images. 16 distortion types are defined, including additive noise, multiplicative noise, spatially correlated noise, high-frequency noise, impulse noise, Gaussian noise, JPEG compression distortion, JPEG2000 compression distortion, local block orientation distortion, contrast distortion, color quantization and jitter, chromatic aberration, color blocking, oversharpening, overexposure, sparse sampling and reconstruction, and 4 distortion intensities. There are a total of 64 combinations of distortion types and intensities. Use these 64 distortion combinations to generate 64 distorted images for each high-resolution image, resulting in a total of 303616 distorted images. Each distorted image and its corresponding high-resolution image form an image pair.

[0090] Step A102: Construct an image restoration network. In some preferred embodiments, the UNet network is used as the image restoration network. The UNet network consists of an encoder and a decoder. The encoder consists of convolution operations and downsampling operations. The convolution structure used is uniformly 3x3 convolution kernel, padding=0, striding=1. The decoder consists of convolution operations and upsampling. Before each convolution operation, skip connections are used to concatenate the encoder features and decoder features.

[0091] Step A103: Train the image restoration network. Take any image pair from the image pairs generated in step A101, which includes a distorted image I and its corresponding high-resolution image. The distorted image I is input into the encoder of the image restoration network constructed in step A102. The encoder outputs a feature map, which is then input into the decoder to output a restored image. Calculate the restored image and high-definition images The distance l1 is used as the loss function L1, and the calculation method is shown in equation (5):

[0092]

[0093] Where W, H, and C represent the restored image, respectively. The width, height, and number of channels are used, and the symbol |·| represents the absolute value. The weights of the image restoration network are updated using the loss function L1.

[0094] Step A201: Collect three open-source IQA datasets, including CSIQ, LIVE, and TID2013. Normalize the image quality scores labeled in these datasets to MOS scores between [0,1]. The CSIQ dataset uses DMOS scores between [0,1], which are converted to MOS scores using 1-DMOS. The LIVE dataset uses DMOS scores between [0,100], which are first converted to MOS scores using 100-DMOS, and then converted to MOS scores between [0,1]. The TID2013 dataset uses MOS scores between [0,10], which are directly converted to MOS scores between [0,1].

[0095] Step A202: Construct an image quality assessment sub-network. First, take the encoder of the trained image restoration network as the feature extractor. Then, connect three sub-network branches after the feature extractor. Each sub-network is responsible for one dataset from CSIQ, LIVE, and TID2013, learning the mapping from image features of that dataset to image quality scores. In some preferred embodiments, this sub-network consists of three 1×1 convolutional operations, where the first two convolutional layers are used to predict the input feature map f. i Visual saliency map A, then f i Performing a point-to-point multiplication with A yields the feature map f. a Finally, f a After passing through a 1×1 convolutional layer and an average pooling layer, the output is the predicted image quality score;

[0096] Step A203: Train the image quality assessment subnetwork by randomly selecting one sample from each of the three IQA datasets collected in step A201. Each sample contains one distorted image. n and an image quality score s n , n (1≤n≤N) represents the index of N IQA datasets, and the distorted image I n The input feature extractor outputs image features, which are then input into the nth sub-network to output the predicted image quality score. calculate and s n The L1 distance is used as the loss function of the nth subnetwork. The calculation method is shown in equation (6):

[0097]

[0098] Where the symbol |·| represents the absolute value, and this loss function is used... The weights of the nth subnetwork are updated, and the weights of the feature extractor are fine-tuned using the sum of the loss functions of the three subnetworks.

[0099] Step A301: Construct the target IQA dataset, collect 200 distorted images, and have 7 scorers label the image quality score for each distorted image. After the scoring is completed, clean the labeled data, remove unreliable labeled data, and finally take the average of all scores for each distorted image as the labeled score for that image. Normalize the labeled score to the same labeling format and score range as in step A201.

[0100] Step A302: Construct an image quality detection network. Take the encoder of the image restoration network trained in step A103 as the first feature extractor, and take multiple sub-networks trained in step A203 as the second feature extractor. Then connect an image quality detection network after the second feature extractor. This network is used to learn the mapping from image features to image quality scores on the target IQA dataset constructed in step A301.

[0101] Step A33: The image quality detection network described in step A302 includes a cross-dataset quality assessment correlation module and a self-attention module, used to mine the correlation of human scoring rules learned on multiple IQA datasets in step A203. Assuming that the feature maps output by the third convolutional layer of the three sub-networks of the second feature extractor are f1, f2 and f3 respectively, in the cross-dataset quality assessment correlation module, firstly, three 1×1 convolutional layers are used to calculate key vectors k1, k2 and k3 for f1, f2 and f3 respectively, and then the correlation of k1, k2 and k3 is calculated pairwise using equation (7):

[0102]

[0103] Among the symbols This represents point-to-point multiplication, c ij The larger the value, the greater the characteristic f. i and f j The greater the correlation between them, the better the effect of equation (8) on f. i and f j Aggregate features with strong correlation:

[0104] f ij =f i +c ij f j ,(i≠j) (8)

[0105] And through equation (9) for f i and f j Separate features with weak correlation:

[0106]

[0107] After calculating the correlation between f1, f2, and f3 pairwise and performing feature aggregation and feature separation, we obtain f 12 f 23 and f 13 ,as well as and Then the two sets of features are input into the self-attention module, with f 12 f 23 and f 13 For example, after entering the self-attention module, the three feature maps are first concatenated along the channels to obtain feature map x, and then passed through three 1×1 convolutional layers in parallel to obtain three feature maps δ(x), φ(x), and τ(x) respectively. Then, the three feature maps are used to calculate the globally strongly correlated feature y using equation (10):

[0108]

[0109] Where Γ(x) represents the number of pixels in the feature maps δ(x), φ(x), and τ(x), and T represents the transpose operation, when the self-attention module inputs... and Perform the same operation to output globally weakly correlated features. Finally, y and The images are stitched together along the channel layers and then passed through several convolutional layers to output the predicted image quality score.

[0110] Step A304: In some preferred embodiments, y and After being stitched along the channel layers, the image passes through seven 1×1 convolutional layers in sequence, with the number of channels being {1024, 512, 256, 128, 64, 32, 1}, to obtain the predicted image quality score.

[0111] Step A303: Train the image quality detection network. Randomly select a sample from the target IQA dataset constructed in step A301. Each sample contains a distorted image I and an image quality score s. The distorted image I is passed through the first and second feature extractors in sequence to output image features. Then, the image features are input into the image quality detection network to output the predicted image quality score. calculate The L1 distance between s and s is used as the loss function L3. This loss function is used to update the weights of the image quality detection network, while the weights of the first and second feature extractors are fine-tuned.

[0112] Beneficial effects of the technical solution of this invention

[0113] (1) The first step of this invention is to pre-train on the image restoration task. On the one hand, this is because image restoration and IQA tasks have many similarities. Both use distorted images as input, and both need to perceive the semantic and distortion information of the image to complete the restoration task and the image quality score prediction task. On the other hand, training for the image restoration task does not require labeled data, which is very easy to obtain. It is only necessary to apply various distortion factors to high-definition images to generate distorted images, thereby obtaining training image pairs. Therefore, generating a large number of distorted-high-definition image pairs and pre-training on the image restoration task can help the IQA model fully learn prior knowledge about image semantics and distortion.

[0114] To address this, we trained two models: one with the first step of image restoration pre-training removed and the other with the first step retained. We used four open-source IQA datasets as target datasets, and the test results on these four datasets are shown in Table 1 (Table 1 compares the experimental results for removing and retaining the first step of image restoration pre-training). The evaluation metrics used were the Spearman Rank Correlation Coefficient (SRCC) and the Pearson Linear Correlation Coefficient (PLCC), commonly used in IQA tasks. It can be seen that the model without image restoration pre-training shows a significant decrease in both metrics across the four IQA datasets. This demonstrates that image restoration pre-training helps improve the prediction accuracy of this model.

[0115] Table 1

[0116]

[0117] (2) In the second step of this invention, several existing IQA datasets were collected for pre-training. However, due to differences in the score labeling format and score range of these datasets (e.g., DMOS with CSIQ set to [0,1], MOS with TID2013 set to [0,10], DMOS with LIVE set to [0,100], etc.), these datasets could not be directly mixed together for training. Therefore, this solution designs multiple independent sub-networks, each learning the scoring rules of a separate IQA dataset. Finally, a cross-dataset quality assessment correlation module is designed to learn the strong and weak correlation features of each sub-network, thereby identifying the commonalities and differences in the scoring rules of these IQA datasets. The learned scoring rules are used as prior knowledge to further improve the performance of the IQA model.

[0118] To address this, we trained two models: one with the second-step multi-branch, multi-dataset joint pre-training removed, and the other with the second-step multi-branch, multi-dataset joint pre-training retained. We used four open-source IQA datasets as target datasets, and evaluated them using SRCC and PLCC metrics. The test results are shown in Table 3 (Table 2 compares the experimental results with and without the second-step multi-branch, multi-dataset joint pre-training). It can be seen that the model without the second-step pre-training shows a significant decrease in both metrics across the four IQA datasets. This demonstrates that the second-step pre-training also helps improve the prediction accuracy and generalization of this model.

[0119] Table 3

[0120]

[0121] In addition, we trained models with and without the cross-dataset quality assessment correlation module, respectively, and the test results are shown in Table 5 (Table 4 compares the experimental results with and without the cross-dataset quality assessment correlation module). It can be seen that the model without this module shows a significant decrease in both metrics across the four IQA datasets. This demonstrates that the cross-dataset quality assessment correlation module plays a crucial role in improving the performance of this model.

[0122] Table 5

[0123]

[0124] A second embodiment of the present invention provides a no-reference image quality detection system based on multi-stage pre-training, comprising a first unit and a second unit;

[0125] The first unit includes a first feature extractor and a second feature extractor connected in sequence, used to extract image features from the input image; the first feature extractor is an encoder in the image restoration network; the second feature extractor is a set of multiple parallel feature extraction sub-networks with consistent structure, which are obtained after overall training using image quality assessment methods based on different IQA datasets;

[0126] The second unit includes an image quality detection network; the image quality detection network is constructed based on cross-dataset quality detection correlation and self-attention mechanism, and obtains image quality detection results based on the image features output by the second feature extractor.

[0127] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process and related descriptions of the system described above can be found in the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0128] It should be noted that the multi-stage pre-training-based no-reference image quality detection system provided in the above embodiments is only an example of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the modules or steps in the embodiments of the present invention can be further decomposed or combined. For example, the modules in the above embodiments can be merged into one module, or further divided into multiple sub-modules to complete all or part of the functions described above. The names of the modules and steps involved in the embodiments of the present invention are only for distinguishing the various modules or steps and are not considered as an improper limitation of the present invention.

[0129] An electronic device according to a third embodiment of the present invention includes: at least one processor; and a memory communicatively connected to at least one of the processors; wherein the memory stores instructions executable by the processor, the instructions being executed by the processor to implement the above-described multi-stage pre-training-based no-reference image quality detection method.

[0130] A computer-readable storage medium according to a fourth embodiment of the present invention is characterized in that the computer-readable storage medium stores computer instructions, which are executed by the computer to implement the above-described method for no-reference image quality detection based on multi-stage pre-training.

[0131] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process and related descriptions of the storage device and processing device described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0132] Specifically, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication component, and / or installed from a removable medium. When the computer program is executed by a central processing unit (CPU), it performs the functions defined in the methods of this application. It should be noted that the computer-readable medium described above in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on a computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0133] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0134] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0135] The terms “first”, “second”, etc., are used to distinguish similar objects, not to describe or indicate a specific order or sequence.

[0136] The term "comprising" or any other similar term is intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus / device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent in such process, method, article, or apparatus / device.

[0137] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A no-reference image quality detection method based on multi-stage pre-training, characterized in that, Includes the following steps: The first feature extractor and the second feature extractor sequentially extract features from the image to be detected to obtain image features. The first feature extractor is the encoder in the image restoration network, and the second feature extractor is a series of parallel feature extraction sub-networks with the same structure, which are trained as a whole based on different IQA datasets using image quality assessment methods. Based on the image features, the detection results are obtained through an image quality detection network; the image quality detection network is constructed based on cross-dataset quality detection correlation and self-attention mechanism. The construction method of the second feature extractor is as follows: Multiple image quality assessment networks with the same network structure are set up in parallel. After being trained on a corresponding number of IQA datasets, the feature extraction part is selected as the second feature extractor. The image quality assessment network is trained using a two-stage weight update method: Each image quality assessment network is trained at level one based on the absolute difference between the output evaluation value and the sample labeled evaluation value as the loss function. Secondary training is performed on multiple image quality assessment networks that are combined in parallel, based on the sum of the loss functions of all image quality assessment networks. The acquisition of image features includes: The image to be detected is input into the first feature extractor to obtain the first image features; The first image features are simultaneously input into each feature extraction sub-network of the second feature extractor to obtain the image features.

2. The no-reference image quality detection method based on multi-stage pre-training according to claim 1, characterized in that, The cross-dataset quality detection correlation calculation includes: calculating... , ... The correlation between pairs of features, and the two sets of features obtained after feature aggregation and feature separation based on the correlation; among them, 、 …… These are the features extracted from the N sub-networks of the second feature extractor.

3. The no-reference image quality detection method based on multi-stage pre-training according to claim 2, characterized in that, , ... Methods for calculating pairwise correlation include: Using N 1×1 convolutional layers respectively to , ... Calculate the key vector , ... ; right , ... The correlation was calculated pairwise to obtain ; in, ，( )； Features and Correlation between them; symbols This indicates a point-to-point multiplication operation.

4. The no-reference image quality detection method based on multi-stage pre-training according to claim 3, characterized in that, The methods for feature aggregation and feature separation are as follows: ，( )； ，( )； in, for and Features after feature aggregation for and Features after feature separation.

5. The no-reference image quality detection method based on multi-stage pre-training according to claim 4, characterized in that, The image quality detection network employs a self-attention mechanism, based on... , Calculate global strong correlation features Global weak correlation characteristics .

6. A no-reference image quality detection system based on multi-stage pre-training, characterized in that, Includes Unit 1 and Unit 2; The first unit includes a first feature extractor and a second feature extractor connected in sequence, used to extract image features from the input image; the first feature extractor is an encoder in the image restoration network; the second feature extractor is a set of multiple parallel feature extraction sub-networks with consistent structure, which are obtained after overall training using image quality assessment methods based on different IQA datasets; The second unit includes an image quality detection network; the image quality detection network is constructed based on cross-dataset quality detection correlation and self-attention mechanism, and obtains image quality detection results based on the image features output by the second feature extractor; The construction method of the second feature extractor is as follows: Multiple image quality assessment networks with the same network structure are set up in parallel. After being trained on a corresponding number of IQA datasets, the feature extraction part is selected as the second feature extractor. The image quality assessment network is trained using a two-stage weight update method: Each image quality assessment network is trained at level one based on the absolute difference between the output evaluation value and the sample labeled evaluation value as the loss function. Secondary training is performed on multiple image quality assessment networks that are combined in parallel, based on the sum of the loss functions of all image quality assessment networks. Image features are obtained, including: The image to be detected is input into the first feature extractor to obtain the first image features; The first image features are simultaneously input into each feature extraction sub-network of the second feature extractor to obtain the image features.

7. An electronic device, characterized in that, include: At least one processor; And a memory communicatively connected to at least one of the processors; wherein the memory stores instructions executable by the processor to implement the multi-stage pre-training-based no-reference image quality detection method according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that are executed by the computer to implement the no-reference image quality detection method based on multi-stage pre-training as described in any one of claims 1-5.

Images