Self-supervised learning based no-reference image quality determination method, system and device

Abstract

The application discloses a kind of based on self-supervised learning's no reference image quality determination method, system and equipment, it is related to image quality evaluation field, the method includes: the image collected is distorted, and the distortion type and distortion degree of image after distortion processing are determined;After distortion processing image is cropped, obtain the first training set that multiple distorted images constitute;Based on self-supervised learning method, the feature extraction network is trained using the first training set, obtain feature extraction model, feature extraction model is used to extract the image distortion feature of input image;The weight in feature extraction model is transferred to quality prediction model using the method of transfer learning, and quality prediction model is the network after projection layer in feature extraction model is replaced by ridge regression model;Quality prediction model is used to predict the quality score of input image;The quality score of the image to be evaluated is predicted using quality prediction model.The application improves the accuracy of image quality prediction.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image quality evaluation, in particular to a no-reference image quality determination method, system and device based on self-supervised learning. BACKGROUND

[0002] With the development of information technology, the method of using digital images as information representation and information communication has been rapidly developed and plays an indispensable role in daily production. In the current digital image processing and communication system, due to the limitations of acquisition errors, data compression, noisy transmission channels and image rendering devices, image signals may experience various forms of distortion and distortion, thereby reducing the quality of the image. For example, in image recognition and target detection tasks, the quality of image acquisition will directly affect the accuracy of recognition and detection; in the field of geographic mapping research, when relevant image data is acquired by a drone or other mapping equipment, the quality of the acquired image also plays a key role in the mapping data.

[0003] A large number of no-reference image quality evaluation methods based on deep learning models have been proposed, although these methods can achieve good results in most scenarios, but these methods all need to use a large number of data sets with real labels to train the model, and most of the available image quality evaluation data sets are too small to effectively train a deep convolutional neural network model from scratch. In view of this, most researchers have begun to use the method of transfer learning, first pre-training the model on some large public data sets, and then fine-tuning the image end-to-end through subjective quality judgment. Although these fine-tuned models have achieved good performance, but since fine-tuning requires carefully selected hyperparameters, excessive fine-tuning can lead to overfitting on the training data, thereby limiting its generalizability. Although this type of model has achieved significant results in the quality evaluation of general synthetic distorted images, but there are still real distorted images in actual application scenarios, and real distorted images often mix multiple distortions or distortions, and usually have no information about the distortion type. SUMMARY

[0004] The purpose of the present application is to provide a no-reference image quality determination method, system and device based on self-supervised learning, which improves the accuracy of image quality prediction.

[0005] To achieve the above purpose, the present application provides the following scheme:

[0006] A no-reference image quality determination method based on self-supervised learning, comprising:

[0007] performing distortion processing on the collected image, and determining the distortion type and distortion degree of the image after distortion processing;

[0008] cropping the distorted processed image to obtain a first training set composed of multiple distorted images;

[0009] training a feature extraction network based on a self-supervised learning method using the first training set to obtain a feature extraction model, the feature extraction model being configured to extract image distortion features of an input image;

[0010] transferring weights in the feature extraction model to a quality prediction model using a transfer learning method, the quality prediction model being a network in which a projection layer in the feature extraction model is replaced by a ridge regression model; the quality prediction model being configured to predict a quality score of an input image;

[0011] predicting a quality score of an image to be evaluated using the quality prediction model.

[0012] Optionally, training a feature extraction network based on a self-supervised learning method using the first training set to obtain a feature extraction model, specifically comprising:

[0013] using contrastive learning as the self-supervised learning method, training a feature extraction network using the first training set to obtain a feature extraction model;

[0014] The loss function used when training the feature extraction network using the first training set is a contrastive loss function, which is a normalized temperature scale cross-entropy loss function.

[0015] Optionally, the feature extraction network comprises an encoder and a projection layer connected in sequence; the encoder is a ResNet50 network with a fully connected layer removed.

[0016] Optionally, a dilated spatial pyramid pooling module is used to bridge the encoder and the projection layer.

[0017] Optionally, a convolutional attention mechanism module is connected after each residual structure of the encoder.

[0018] Optionally, the collected image is distorted processed, and the distortion type and distortion degree of the distorted processed image are determined, specifically comprising:

[0019] The collected image is used as a full-resolution image, the full-resolution image is distorted processed, and the distortion type and distortion degree of the distorted processed image corresponding to the full-resolution image are determined.

[0020] The full-resolution image is down-sampled along two dimensions to obtain a half-resolution image, the half-resolution image is distorted processed, and the distortion type and distortion degree of the distorted processed image corresponding to the half-resolution image are determined.

[0021] Optionally, the ridge regression model is a ridge regression classifier trained according to a second training set; the sample data in the second training set comprises input data and label data, the input data is a sample image, and the label data is an image quality score.

[0022] Optionally, the image to be evaluated comprises a portrait photo.

[0023] The application discloses a kind of based on self-supervised learning's no reference image quality determination system, comprising:

[0024] Image distortion processing module, for the distortion processing to the image collected, and the distortion type and distortion degree of image after distortion processing are determined;

[0025] First training set construction module, for the distortion processing image is cropped, and a plurality of distortion images constitute first training set are obtained;

[0026] Feature extraction network training module, for based on self-supervised learning method, using first training set trains feature extraction network, obtains feature extraction model, and the feature extraction model is used to extract the image distortion feature of input image;

[0027] Quality prediction model determination module, for using the method of transfer learning to transfer the weight in the feature extraction model to quality prediction model, the quality prediction model is the network after the projection layer in the feature extraction model is replaced by ridge regression model;The quality prediction model is used to predict the quality score of input image;

[0028] Quality prediction model application module, using the quality prediction model to carry out quality score prediction to image to be evaluated.

[0029] The application further discloses an electronic device comprising a memory and a processor, the memory is used to store computer programs, and the processor runs the computer programs to make the electronic device execute the self-supervised learning's no reference image quality determination method according to the application.

[0030] According to the specific embodiments provided by the application, the following technical effects are disclosed:

[0031] The application realizes the feature extraction of the image of different distortion types by the distortion of different types and different degrees of the image collected to constitute first training set and training feature extraction network by first training set, so as to improve the accuracy of image quality prediction. BRIEF DESCRIPTION OF DRAWINGS

[0032] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 A schematic flowchart of a method for determining the quality of a no-reference image based on self-supervised learning, provided in an embodiment of the present invention;

[0034] Figure 2 A schematic diagram of the acquired images and corresponding distorted images provided for embodiments of the invention;

[0035] Figure 3 A schematic diagram illustrating the training principle of the feature extraction network provided in the embodiments of the invention;

[0036] Figure 4 A schematic diagram of the feature extraction network structure provided for an embodiment of the invention;

[0037] Figure 5 A schematic diagram of the quality prediction model structure provided for an embodiment of the invention;

[0038] Figure 6 This is a schematic diagram of a no-reference image quality determination system based on self-supervised learning, provided as an embodiment of the present invention. Detailed Implementation

[0039] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0040] The purpose of this invention is to provide a method, system, and device for determining the quality of no-reference images based on self-supervised learning, which improves the accuracy of image quality prediction.

[0041] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0042] Example 1

[0043] like Figure 1 As shown in the figure, this embodiment provides a method for determining the quality of a no-reference image based on self-supervised learning, which specifically includes the following steps:

[0044] Step 101: Perform distortion processing on the acquired image and determine the type and degree of distortion of the image after distortion processing.

[0045] The acquired images are processed using a computer to remove distortion, and then categorized by distortion type; d i , i∈{1,…,D} and the degree of distortion l ij j∈{1,…,L i The images are classified to obtain the distorted images. d i Indicates the i-th distortion type, l ij L represents the degree of distortion of the j-th distortion type of the i-th distortion type, D represents the number of distortion types, and L represents the degree of distortion of the j-th distortion type of the i-th distortion type. i This represents the number of distortion levels corresponding to the i-th distortion type.

[0046] Distortion types include Still Image Compression (JPEG), JPEG2000 (JP2K), White Noise (WN), Gaussian Blur (GB), and Fast Fading Rayleigh (FF). Distortion types such as... Figure 2 As shown, Figure 2 The first column of images is the reference image, i.e. the captured image. The second column is the JPEG format corresponding to the captured image. The third column is the JP2K format corresponding to the captured image. The fourth column is the Gaussian blurred image corresponding to the captured image.

[0047] Step 102: Crop the distorted images to obtain the first training set consisting of multiple distorted images.

[0048] In step 102, the distorted image dataset X is constructed by cropping the distorted image to a size of 256×256.

[0049] After cropping the distorted image, let X = {x1, x2, x3, ..., x...} i-1 ,x i ,···,x n}, where x i Let be the i-th distorted image, and n be the total number of distorted images.

[0050] The input samples (distorted images) are randomly cropped to a fixed M×M size. The cropped version inherits the same distortion category as the original. It is assumed that the distortion category remains almost unchanged and is unaffected by the cropping operation. For each input image, two random crops are generated after processing, one at full resolution and the other at half-scale resolution. When the original image size is smaller than the cropped size M×M, the entire image is zero-padding to maintain the same resolution. Cropping provides a batch of images at a fixed resolution.

[0051] Step 103: Based on the self-supervised learning method, a feature extraction network is trained using the first training set to obtain a feature extraction model, which is used to extract image distortion features of the input image.

[0052] Step 103 specifically includes:

[0053] Contrastive learning is used as the self-supervised learning method. The feature extraction network is trained using the first training set to obtain the feature extraction model. That is, X is fed into the feature extraction network to learn image distortion features, and the distortion type and distortion degree are used as class labels.

[0054] Before training the feature extraction network using the first training set, training parameters are pre-set, including the number of iterations, optimizer, learning rate, and data strength ratio.

[0055] When training the feature extraction network using the first training set, the loss function used is the contrastive loss function, which is the normalized temperature-scale cross-entropy loss function.

[0056] like Figure 3 As shown, during the training of the feature extraction network, X is input into the network and, through an auxiliary task, learns a representation that can classify images into distortion-related classes. Images with similar distortion levels are grouped closer together, while images with significantly different distortion levels are pushed further apart. This is achieved using a cross-entropy objective function during training. In the feature extraction network training, prior knowledge of synthetic distortion (distortion type and distortion degree) is used as class labels, which measures the similarity between each pair of distorted images in the batch. Images with similar distortion levels are treated as positive samples, bringing them closer to the original images, while images with significantly different distortion levels are treated as negative samples, pushing them further away. This pairwise loss calculation is a key feature distinguishing it from traditional cross-entropy loss. The original acquired image x is processed by distortion type d... i , i∈{1,…,D} and distortion level l ij j∈{1,…,L i Distortion is applied to obtain a distorted image. D and L i These represent the number of distortion types and the degree of distortion, respectively. The model's task is to identify d. i and l ij This task can be transformed into having The classification problem is a class (total number of distortion levels + one original image). Therefore, for a self-built contrastive learning model, it is only necessary to learn the representation that can distinguish the image quality categories, and it is not necessary to learn the specific quality score representation of a particular image. In order to learn the classification representation, the image representation is extracted by the encoder, and then the contrastive loss method is used to learn the representation. For the similarity between a pair of representations, this model uses the dot product to measure, as shown in formula (1). The contrastive loss function uses normalized temperature scale cross-entropy (NT-Xent), and its formula is for image x i The definition is shown in formula (2).

[0057]

[0058]

[0059] in, Let represent the dot product of image representation a and image representation b, T denote the transpose, where N is the number of images appearing in the batch, l is the indicator function, τ is the temperature parameter, and P(i) is a function containing the relationship between x and y. i The set of indices of distorted images belonging to the same class, where |P(i)| is the cardinality of P(i), and z i Represents the distorted image x i The corresponding eigenvector representation, z j Represents the distorted image x j The corresponding feature vector representation. In the model design, the prior knowledge of synthetic distortion is used as the class label. Objective function (normalized temperature-scaled cross-entropy function). It measures the similarity between each pair of distorted images in the batch processing, taking distorted images with similar distortion levels as positive samples and bringing them closer to the samples, and distorted images with large differences in distortion levels as negative samples and moving them further away from the samples. This pairwise loss calculation is a key feature that distinguishes it from the traditional cross-entropy loss.

[0060] For naturally distorted images, the contrast loss is expressed as:

[0061]

[0062] in, This represents the contrast loss when a naturally distorted image is input into a feature extraction network.

[0063] and Total contrast loss representation for:

[0064]

[0065] Figure 3In this context, f() represents the encoder, g() represents the projection layer, and UGC represents the natural distortion image.

[0066] To better learn feature representations of images at multiple scales, this embodiment uses two scales of input images: full resolution and half resolution. Images themselves possess multi-scale characteristics, and correspondingly, image distortion is also multi-scale; therefore, image distortion manifests differently at different scales. For perceived image quality, local details and global features have a significant impact. Early image quality assessment models attempted to simulate the function of the brain's front-end visual processing by using multi-scale representations to predict quality. Image quality assessment models based on convolutional neural networks, using multi-scale features, can significantly improve efficiency in capturing visual quality.

[0067] Step 101 specifically includes:

[0068] The acquired image is used as a full-resolution image. Distortion processing is performed on the full-resolution image, and the distortion type and degree of distortion of the image after distortion processing are determined.

[0069] A half-resolution image is obtained by downsampling the full-resolution image along two dimensions. The half-resolution image is then subjected to distortion processing, and the distortion type and degree of the distorted image corresponding to the half-resolution image are determined.

[0070] In the process of downsampling the full-resolution image along two dimensions to obtain a half-resolution image, the original aspect ratio of the distorted image is preserved during the resizing operation, because modifying this ratio would affect the quality of the base image. To avoid aliasing artifacts, a Gaussian low-pass aliasing filter is used to pre-filter the full-resolution image before downsampling. Different color spaces are used to extract complementary quality information that may exist in different neighborhoods. This method can improve the model's ability to perceive image details and global information, and can better handle image data in different neighborhoods. In addition, a bandpass transform and Local Mean-Subtraction (MS) are used to obtain quality features. Bandpass transform is a commonly used image enhancement technique that aims to highlight information in certain frequency ranges of an image while suppressing information in other frequency ranges. Local mean subtraction is also a form of bandpass transform, which enhances the high-frequency information of the image by averaging the neighborhood around each pixel and then subtracting the value of that pixel from the average.

[0071] The frequency response of an ideal low-pass filter is shown in equation (3).

[0072]

[0073]

[0074] Where H(u,v) represents an ideal low-pass filter, (u,v) represents a frequency point, D(u,v) represents the distance between (u,v) and the center of the frequency domain, D0 is the cutoff frequency, and D0 controls the bandwidth of the filter. The main drawback of an ideal low-pass filter is that its frequency response has an infinitely large band-stop region. Therefore, the filter will produce large amplitude and phase distortion at the cutoff frequency. The frequency response of a Gaussian low-pass filter is shown in equation (5).

[0075]

[0076] Here, σ is the standard deviation of the Gaussian low-pass filter, which controls the filter's bandwidth. Unlike an ideal low-pass filter, a Gaussian low-pass filter exhibits a smoother transition in its frequency response. Therefore, it does not produce severe amplitude and phase distortion at the cutoff frequency.

[0077] In step 101, distortion processing can construct an auxiliary task for contrastive learning. The goal of the auxiliary task is to learn a representation that can distinguish images into distortion-related classes, and to bring images with similar distortion levels closer together and push images with different distortion levels further apart.

[0078] like Figure 4 As shown, the input image has dimensions of H×W×3, where H represents the image height and W represents the image width. The feature extraction network includes an encoder and a projection layer connected in sequence; the encoder is a ResNet50 network with fully connected layers removed. After processing by the encoder module, the distorted image can be converted into a set of compact feature vector representations. These vectors contain important feature information of the image. To reduce the dimensionality of the high-dimensional feature vectors generated by the encoder, the projection layer uses a multilayer perceptron. Since the encoder removes the fully connected layers, the projection layer can reduce the dimensionality, lowering the high-dimensional feature vectors to the required dimension. The projection layer uses a multilayer perceptron because it can learn complex nonlinear mapping relationships, thus better preserving the important information in the original feature vectors. Figure 4 In this context, conv kernel represents the convolution kernel, and Rate represents the dilation rate.

[0079] The Atrous Spatial Pyramid Pooling (ASPP) module bridges the encoder and the projection layer, increasing the model's receptive field without altering the overall model architecture. This allows for better capture of image details, thereby improving the model's accuracy in judging image quality. The ASPP module uses multiple dilated convolution kernels to convolve the feature maps, obtaining features with different receptive fields. These feature maps are then combined in parallel and finally fused to generate the final feature map.

[0080] The ASPP module consists of a convolutional layer and a pyramid pooling layer, with the addition of adaptive mean pooling. The pyramid pooling layers are composed of multiple dilated convolutional pooling layers of varying sizes and dilation rates. By setting different dilation factors, multi-scale features at different scales can be obtained. The key difference between dilated pointwise convolution and ordinary pointwise convolution lies in the dilation rate setting. The dilation rate, through the setting of padding and dilation values, controls the number of convolutional operation intervals. Using dilated convolution allows for setting multiple dilation rates without increasing network depth or the number of parameters, enabling the network to obtain receptive fields at different scales and thus acquire feature information at multiple scales.

[0081] The actual size of the convolution kernel in dilated convolution can be calculated using equation (6).

[0082] K = k + (k-1)(rate-1) (6)

[0083] Where k is the kernel size and r is the dilation rate. When the kernel size is 3, the original receptive region is 3×3; when the dilation rate is set to 3, the receptive region is 7×7; and when the rate is set to 7, the receptive region is 15×15. However, when the dilation rate increases, although the receptive region increases, the kernel loses some detailed information between points when extracting features due to the intervals in the convolution operation. Moreover, the feature map obtained by using only a single layer of dilated convolution will have discontinuous information. Therefore, multiple dilated convolution layers with different dilation rates are used to avoid the problem of information loss.

[0084] Dilated convolutions significantly enhance the detail extraction capabilities of features. When dealing with target distortions at different scales, increasing the receptive field can better capture information from areas of significant distortion, and dilated convolutions achieve this by expanding the receptive field of the entire model. Because large distortions cover a large area, a receptive field as large as possible is needed to perceive distortion information over long distances, which is crucial for acquiring quality features from large-area distortions. While pooling layers can expand the receptive field, they typically compress feature maps and reduce image resolution, potentially leading to information loss and blurring. In contrast, dilated convolutions better preserve the feature information of the original image because they expand the receptive field without reducing the feature map, thus better capturing large-scale image structures and details, especially effective for extracting features from targets with significant distortions.

[0085] When processing images with significant distortion, feature extraction networks need to possess a certain degree of context awareness. By using dilated convolutions instead of traditional compact convolution kernels, the gaps between the kernels necessitate padding with weights to ensure the convolution operation fully considers the relationships between these locations and surrounding pixels. Padding with zeros does not affect the original weight calculations during network training. The number of zeros determines the receptive field size of the kernel; more padding results in a larger receptive field. By combining dilated convolutions of different sizes, the receptive field can be expanded, better capturing the contextual relationships in the image and enhancing the network's feature representation capabilities.

[0086] Traditional image quality assessment networks typically use pooling layers to increase the network's receptive field, but this reduces image resolution, which needs to be maintained during image quality evaluation. Using a feature extraction network that incorporates ASPP (Application Performance Profile) can extract richer semantic information and better extract information about different types of distortion.

[0087] Therefore, this embodiment takes advantage of the fact that residual networks have stronger expressive power and generalization performance. By using the hollow spatial pyramid pooling module as a bridging module between the encoder and the projection layer network, the receptive field of the model can be increased without changing the overall model architecture. This allows for better capture of detailed information in the image, thereby improving the model's accuracy in judging image quality.

[0088] Each residual structure of the encoder is followed by a Convolutional Block Attention Module (CBAM). CBAM enhances the extraction of important information, prevents important quality features from being ignored, strengthens attention to key information, suppresses the learning of irrelevant information, and improves the accuracy of image quality assessment. CBAM is a lightweight convolutional block model with plug-and-play characteristics, which can be integrated into any convolutional neural network to improve model performance through co-training. Its main function is to add attention mechanisms in the channel and spatial dimensions, helping the network to better focus on key features in the image, thereby improving the discriminative and generalization abilities of features.

[0089] During the training of the feature extraction network, the Channel Attention Module (CAM) generates channel attention maps based on the channel dimension, giving each channel a different weight value. Throughout the network, channels with larger weight values ​​receive more attention, thereby enhancing the representation extraction performance of the network. Channel Attention M c (F) The calculation formula is shown in equation (7):

[0090]

[0091] The CBAM module selects max pooling and average pooling to process the input feature F in order to obtain local maxima. and average These two feature maps are then input into a shared multilayer perceptron network for computation, which consists of multiple perceptrons. After computation by the multilayer perceptron, two feature vectors are obtained. These two vectors are added together and then processed using the sigmoid activation function. The result is the weight values ​​for different channels. These weight values ​​are used to calculate the channel attention map, so that the network focuses on important features in the channel dimension of the convolutional feature map. In formula (7), σ represents the sigmoid activation function, W0∈R Cr×C ,W1∈R C×Cr W0 and W1 are the weights of the multilayer perceptron (MLP), C represents the number of channels, r represents the channel reduction rate, and R... Cr×C R C×Cr Both are data dimensions. AugPool() represents average pooling, and MaxPool() represents max pooling.

[0092] In image quality assessment, attention mechanisms are used to enhance the neural network's focus on regions with local distortions, thereby improving the accuracy of evaluating natural distortions. The computation process of the spatial attention module is as follows: Figure 4As shown: The convolutional output of the previous layer is fed into the spatial attention module, and then pooling is used to obtain the maximum and average values ​​in each local region. These two values ​​are then fed into the standard convolutional layer for convolution, resulting in a set of feature maps with an attention mechanism. These feature maps will receive different weights in both dimensions to differentiate between different feature map regions. The calculation process is shown in equation (8):

[0093]

[0094] Similar to the calculation process of the channel attention mechanism, two result feature maps are obtained after calculation. and Finally, after convolution, the final spatial attention map M is obtained. s (F).

[0095] Max pooling is effective for encoding salient features because it captures the most salient features within a local region, while average pooling is better suited for encoding global features because it smooths the information across the entire feature map. Therefore, both max pooling and average pooling are used in the computation in both modules.

[0096] Step 104: The weights in the feature extraction model are transferred to the quality prediction model using transfer learning. The quality prediction model is a network in which the projection layer in the feature extraction model is replaced with a ridge regression model. The quality prediction model is used to predict the quality score of the input image.

[0097] The ridge regression model is a ridge regression classifier trained on a second training set; the sample data in the second training set includes input data and label data, the input data being sample images and the label data being image quality scores.

[0098] like Figure 5 As shown, in the quality prediction model, the encoder output is used as an image representation. A regularized linear regressor—ridge regression—is trained on the frozen encoder network. The classifier is typically a simple linear model, such as logistic regression or support vector machines. The main idea of ​​the linear evaluation protocol is that the learned representation should contain enough information for the linear classifier to perform well on new tasks. In the image quality assessment task, the generated representation is input into the ridge regression model for linear regression to obtain a predicted quality score. Figure 5 The upper dashed box represents the prior task, and the lower dashed box represents the target task.

[0099] Step 105: Use the quality prediction model to predict the quality score of the image to be evaluated.

[0100] If X from step 102 is input into the quality prediction model, the output evaluation result R = {r1, r2, r3, ... r...} i-1 ,r i ,···,r n}, where r i For x i The corresponding rating results.

[0101] The images to be evaluated include photographs of people.

[0102] The following specific example illustrates a self-supervised learning-based method for improving the quality of no-reference images according to the present invention.

[0103] The execution process of this invention first involves processing the acquired images using a computer to obtain distorted images, and then classifying them according to different types and degrees of distortion to obtain image distortion.

[0104] Furthermore, the distorted images are cropped to obtain the image quality assessment database (first training set). The specific steps are as follows:

[0105] Step S1: Crop the image to a fixed size of 256×256 to ensure that the input image to the model maintains a uniform size.

[0106] Step S2: Two scales are used: the original resolution and a half-scale resolution obtained by downsampling by a factor of two along both dimensions. The original aspect ratio of the image is preserved during the resizing operation because modifying this ratio would affect the quality of the base image. To avoid aliasing artifacts, a Gaussian low-pass aliasing filter is used for pre-filtering before downsampling.

[0107] Step S3: Perform a random cropping operation on the input samples to fix them. A simplified assumption is made that the cropped version inherits the same distortion category as the original version. Although the cropped version does not need to represent the same perceptual quality as the original image, it is assumed that the distortion category remains almost unchanged and is unaffected by the cropping operation. For each input image, two random crops are generated after processing, one at full resolution and the other at half-scale resolution.

[0108] Furthermore, the first training set is input into a feature extraction network based on feature enhancement to obtain the optimal feature extraction model. Figure 4 The network structure of the feature extraction network is shown, and the specific steps are as follows:

[0109] Step S4: Input the original image [256,256,3] into the backbone feature extraction network. It first goes through the first stage (layer 0), consisting of a 7×7 convolutional layer with a stride of 2, followed by Batch Normalization (BN) and ReLU, plus a 3×3 max-pooling layer with a stride of 2, resulting in a feature map of [64,64,64]. Here, BN represents batch normalization, and ReLU represents the ReLU activation function. The second stage consists of residual blocks composed of varying numbers of bottleneck layers. These bottleneck layers are of two types: identity blocks and convolutional blocks, used to address the input-output size mismatch. The first residual block contains three bottleneck layers: one convolutional block and two identity blocks, with no downsampling. The second, third, and fourth residual blocks consist of 4, 6, and 3 bottleneck layers, respectively. The first bottleneck layer in these three layers is a convolutional residual block with a stride of 2. This is because each layer needs to downsample the feature image size, and this downsampling process occurs in the first bottleneck layer. During this process, the feature image size changes. The other bottleneck layers all use identity residual blocks, so the dimension of the feature image remains unchanged. The final feature image is [8, 8, 2048].

[0110] Step S5: Add the CBAM attention mechanism to each layer structure. In each residual structure, the feature map is first convolved with convolutions of size 1×1, 3×3 and 1×1. After that, the CBAM module is added without changing the number of channels. By adding the CBAM attention mechanism, each residual block can pay more attention to channel and spatial location information, thereby enhancing the feature representation ability.

[0111] Step S6: After the feature network outputs the [8,8,2048] feature map, it is input into the spatial pyramid pooling layer, which consists of a convolution and a pyramid pooling layer, with an adaptive mean pooling layer added. The multiple layers of the pyramid pooling layer are composed of multiple dilated convolution pooling layers of different sizes and dilation rates. By setting different dilation factors, multi-scale features at different scales are obtained. The first pooling layer is a 3×3 convolution with a dilation rate of 1, the second pooling layer is a 3×3 convolution with a dilation rate of 6, the third pooling layer is a 3×3 convolution with a dilation rate of 12, and the fourth pooling layer is a 3×3 convolution with a dilation rate of 18. Then, the four feature maps are fused through a 1×1 convolution to obtain the final feature map.

[0112] Step S7: Input the feature map obtained from the previous layer into the projection layer module composed of a multilayer perceptron to reduce the representation dimension and obtain the final feature map.

[0113] Step S8: The final feature map is calculated using contrastive loss. The goal of this is to obtain similar representations through the encoder network, so that images belonging to the same category have similar representations. The similarity between a pair of representations is measured by the dot product in this model. The contrastive loss function used is normalized temperature-scaled cross-entropy (NT-Xent).

[0114] Step S9: After the feature extraction network is trained, the projection layer network is removed, and the output of the encoder network is used as the image representation. A regularized linear regressor—ridge regression—is trained on the frozen encoder network. Its advantage lies in its adaptability to high-dimensional data and multicollinearity. In high-dimensional data, using ridge regression can effectively avoid overfitting. This is similar to the linear evaluation protocol used to assess the classification accuracy of self-supervised models. This protocol uses a pre-trained model to generate representations and feeds them into a linear classifier for fine-tuning for classification tasks. This classifier is typically a simple linear model, such as logistic regression or support vector machines. The main idea of ​​the linear evaluation protocol is that the learned representations should contain enough information for the linear classifier to perform well on new tasks. In the image quality assessment task, the generated representations are fed into the ridge regression model for linear regression to obtain a predicted quality score.

[0115] Step S10: According to the pre-set iteration rounds, the model will be evaluated once in each round, and the optimal model will be selected as the detection model (quality prediction model).

[0116] Example 2

[0117] like Figure 6 As shown, this embodiment provides a no-reference image quality determination system based on self-supervised learning, comprising:

[0118] The image distortion processing module 201 is used to perform distortion processing on the acquired image and determine the distortion type and degree of distortion of the image after distortion processing.

[0119] The first training set construction module 202 is used to crop the distorted images to obtain a first training set composed of multiple distorted images.

[0120] The feature extraction network training module 203 is used to train the feature extraction network using a first training set based on a self-supervised learning method to obtain a feature extraction model, which is used to extract image distortion features of the input image.

[0121] The quality prediction model determination module 204 is used to transfer the weights in the feature extraction model to the quality prediction model using a transfer learning method. The quality prediction model is a network after replacing the projection layer in the feature extraction model with a ridge regression model. The quality prediction model is used to predict the quality score of the input image.

[0122] The quality prediction model application module 205 uses the quality prediction model to predict the quality score of the image to be evaluated.

[0123] Example 3

[0124] This embodiment provides an electronic device, including a memory and a processor. The memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to perform the self-supervised learning method for determining the quality of a referenceless image according to Embodiment 1.

[0125] This embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the self-supervised learning method for determining the quality of a referenceless image as described in Embodiment 1.

[0126] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0127] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for determining the quality of a no-reference image based on self-supervised learning, characterized in that, include: The acquired images are subjected to distortion processing, and the type and degree of distortion of the processed images are determined. The distorted images are cropped to obtain the first training set consisting of multiple distorted images; Based on the self-supervised learning method, a feature extraction network is trained using the first training set to obtain a feature extraction model, which is used to extract image distortion features of the input image. The weights in the feature extraction model are transferred to the quality prediction model using transfer learning. The quality prediction model is a network in which the projection layer of the feature extraction model is replaced with a ridge regression model. The quality prediction model is used to predict the quality score of the input image. The quality prediction model is used to predict the quality score of the image to be evaluated; The feature extraction network includes an encoder and a projection layer connected in sequence; the encoder is a ResNet50 network with the fully connected layers removed; a dilated spatial pyramid pooling module is used to bridge the encoder and the projection layer; each residual structure of the encoder is followed by a convolutional attention mechanism module; The acquired images are subjected to distortion processing, and the type and degree of distortion of the processed images are determined, specifically including: The acquired image is used as a full-resolution image. Distortion processing is performed on the full-resolution image, and the distortion type and degree of distortion of the image after distortion processing are determined. The full-resolution image is downsampled along two dimensions to obtain a half-resolution image. The half-resolution image is then distorted while retaining the original aspect ratio. The distortion type and degree of distortion of the distorted image corresponding to the half-resolution image are then determined.

2. The method for determining the quality of a no-reference image based on self-supervised learning according to claim 1, characterized in that, Based on a self-supervised learning method, a feature extraction network is trained using the first training set to obtain a feature extraction model, which specifically includes: Contrastive learning is used as the self-supervised learning method, and the feature extraction network is trained using the first training set to obtain the feature extraction model. When training the feature extraction network using the first training set, the loss function used is the contrastive loss function, which is the normalized temperature scale cross-entropy loss function.

3. The method for determining the quality of a no-reference image based on self-supervised learning according to claim 1, characterized in that, The ridge regression model is a ridge regression classifier trained on a second training set; the sample data in the second training set includes input data and label data, the input data being sample images and the label data being image quality scores.

4. The method for determining the quality of a no-reference image based on self-supervised learning according to claim 1, characterized in that, The images to be evaluated include photographs of people.

5. A no-reference image quality determination system based on self-supervised learning, characterized in that, The self-supervised learning-based no-reference image quality determination system applies the self-supervised learning-based no-reference image quality determination method of claim 1, wherein the self-supervised learning-based no-reference image quality determination system comprises: The image distortion processing module is used to process the acquired images and determine the type and degree of distortion of the processed images. The first training set construction module is used to crop the distorted images to obtain a first training set consisting of multiple distorted images; The feature extraction network training module is used to train the feature extraction network based on the self-supervised learning method using the first training set to obtain the feature extraction model, which is used to extract the image distortion features of the input image. A quality prediction model determination module is used to transfer the weights in the feature extraction model to the quality prediction model using a transfer learning method. The quality prediction model is a network in which the projection layer in the feature extraction model is replaced with a ridge regression model. The quality prediction model is used to predict the quality score of the input image. The quality prediction model application module uses the quality prediction model to predict the quality score of the image to be evaluated.

6. An electronic device, characterized in that, The device includes a memory and a processor, the memory being used to store a computer program, and the processor running the computer program to cause the electronic device to perform a self-supervised learning method for determining the quality of a referenceless image according to any one of claims 1 to 4.

Images