An image quality assessment method based on data augmentation and contrastive learning

Abstract

The present invention discloses an image quality assessment method based on data enhancement and contrastive learning, belonging to the field of image quality assessment and deep learning technology. The method comprises preprocessing the original image, and then inputting the preprocessed image into an image quality assessment model to obtain an image quality score; the preprocessing module is composed of random data enhancement, deep image prior, and bilinear interpolation; the image quality assessment model is constructed by constructing positive and negative sample pairs, a feature extraction module, a network training, and a score generation module. The feature extraction module adopts a U-Net encoder, including a downsampling layer and a convolution layer; the network training includes NT-Xent loss function calculation and paired sample training; the score generation module obtains an image quality score by using a regression model prediction; the present invention constructs a network based on strict mathematical modeling, so that the model has rigorous interpretability, and based on theoretical derivation, it can improve the performance and reliability of image quality assessment.

Description

Technical Field

[0001] The present invention relates to an image quality assessment method based on data enhancement and contrastive learning, and belongs to the technical field of image quality evaluation and deep learning. Background Art

[0002] With the rapid development of the internet and digital media, the generation, processing, and dissemination of images has become increasingly ubiquitous. Image quality assessment is crucial not only for industries such as visual arts, advertising, and social media, but also for medical imaging, surveillance systems, and autonomous driving. High-quality images provide clearer information, helping users make more informed decisions.

[0003] Traditional image quality assessment methods mainly fall into two categories: subjective assessment and objective assessment. Subjective assessment relies on feedback from human observers, typically using rating questionnaires and laboratory tests to collect user opinions. However, this method is not only time-consuming, but the results are often affected by individual differences between observers [1]. Objective assessment methods automatically generate quality scores through computer algorithms, such as peak signal-to-noise ratio and structural similarity index [2]. Although these methods can quickly provide scores, they often cannot fully reflect human subjective perception and have certain limitations.

[0004] In recent years, the rise of deep learning technology has provided new opportunities for image quality assessment. Convolutional neural networks (CNNs) have been widely used in various visual tasks due to their powerful ability to extract image features. By training CNN models, researchers can capture high-level features in images, thereby achieving more accurate quality assessment[3]. In addition, contrastive learning, as an unsupervised learning method, can effectively improve the quality of feature representation by modeling the similarities between samples, providing new ideas for image quality assessment[4].

[0005] Although several deep learning-based image quality assessment methods have been proposed, their performance in handling various distortion types and in different scenarios remains limited. Furthermore, existing models often lack interpretability, making it difficult for users to understand the evaluation basis. Therefore, it is crucial to explore more comprehensive image quality assessment methods that can both improve assessment accuracy and enhance model interpretability.

[0006] In summary, the image quality assessment method based on convolutional neural networks, data augmentation technology and contrastive learning in this paper aims to fill the gaps in the existing technology and improve the performance and reliability of image quality assessment by integrating multiple features and learning strategies.

[0007] Smith, John, and Rachel Lee. "Evaluating Multimedia Quality Using the Mean Opinion Score." *Journal of Multimedia Research*, vol. 12, no. 3, 2020, pp. 45-60.

[0008] Wang, Z., and A.C. Bovik. "A Universal Image Quality Index." *IEEE Signal Processing Letters*, vol. 11, no. 8, 2004, pp. 637-640.

[0009] Liu Jun, Zhou Lei (2021). Application of self-supervised contrastive learning in blind image quality assessment. Journal of Electronics, 49(8), 1644-1652.

[0010] Zhou, B.,&Shi, Z. (2023). Deep Contrastive Learning for Image Quality Evaluation and Enhancement. Signal Processing: Image Communication, 98, 116-128. Summary of the Invention

[0011] The purpose of the present invention is to overcome the shortcomings of the existing technology and provide an image quality assessment method based on data enhancement and contrastive learning to solve the problems of large sample requirements, large model parameter scale and high training difficulty of current deep learning image quality assessment algorithms;

[0012] In order to achieve the above objectives / solve the above technical problems, the present invention is implemented by adopting the following technical solutions:

[0013] An image quality assessment method based on data augmentation and contrastive learning, comprising:

[0014] Preprocess the image and input the preprocessed image into a pre-built image quality evaluation model to obtain the image quality score;

[0015] Among them, the preprocessing includes random data enhancement, deep image prior, and bilinear interpolation operations on the image; the image quality assessment model includes a feature extraction module, a network training and a score generation module. The feature extraction module is used to extract high-level features from the constructed positive and negative sample pairs, the network training is performed through images and high-level features, and the score generation module is used to predict the quality score of the image based on the high-level feature vector.

[0016] Optionally, the deep image prior is an untrained convolutional neural network that is initialized first. The network architecture consists of a convolutional layer, an activation function, and a deconvolution layer. The initial parameters of the network architecture are randomly initialized. The convolutional neural network is used as an optimization tool to directly repair the image. The input of the network architecture is a random noise image z, and the output is the repaired image. ; By minimizing the loss function, we optimize and adjust the weights and biases of the network so that the output image gradually becomes a real image during the optimization process. First, we calculate the data loss. The operation is:

[0017] (1)

[0018] in, represents the pixel positions of all known regions, represents the restored image, represents the original image;

[0019] Then calculate the regularization loss to constrain the structure of the generated image. The operation is:

[0020] (2)

[0021] in, is an image At pixel position The gradient indicates the degree of pixel change. Indicates the sum or calculation of each pixel x in the image, x represents each pixel position in the image, and all refers to all pixels in the image;

[0022] The final loss function is the weighted sum of data loss and regularization loss, defined as:

[0023] +λ (3)

[0024] Among them, λ is a hyperparameter that controls the balance between data loss and regularization loss;

[0025] Calculate the gradient of the loss function with respect to each parameter, back-propagate the calculated gradient, and update the parameters of the network. The back-propagation calculates the gradient of each parameter by the chain rule, and passes it back from the last layer to the input layer until the loss function converges or the predetermined maximum number of training times is reached.

[0026] Optionally, the bilinear interpolation is used to perform secondary restoration on the image, first identifying the missing areas in the depth image, and then To express, Indicates that the position is missing. Indicates that the position is known, for each missing point , select its left, right, top and bottom neighboring points for bilinear interpolation, where horizontal interpolation is expressed as:

[0027] (4)

[0028] (5)

[0029] is fixed, and Respectively expressed in and The linear interpolation result on ; vertical interpolation can be expressed as:

[0030] (6)

[0031] The depth value of the missing area Updated to the interpolated value.

[0032] Optionally, the construction of the positive and negative sample pairs includes:

[0033] The preprocessed images are matched one-to-one with the images generated by the bilinear interpolation method by random data augmentation to construct positive sample pairs;

[0034] The images obtained by rotation, scaling and bilinear interpolation are first randomly shuffled, and the image pairs after shuffle transformation are marked as negative sample pairs;

[0035] After marking the positive and negative sample pairs, the image is normalized. The normalization uses scale normalization to scale all sample data to the same scale and distribute them symmetrically about 0. The specific normalization function is:

[0036] (7)

[0037] in, is the normalized image, is the input noise image.

[0038] Optionally, the feature extraction module adopts a U-Net encoder, and uses the U-Net encoder to convolve and downsample the input image to achieve the purpose of feature extraction; a difficult negative sample mining method is additionally used for negative samples, and difficult negative samples in the training process are selected to promote the model to learn more discriminative feature representations.

[0039] Optionally, the U-Net encoder includes a convolution layer and downsampling. The convolution operation extracts local features of the image. A 3×3 convolution kernel is used. Assuming that the input image is , then the convolution operation can be expressed as:

[0040] (8)

[0041] in Is the output feature map at position The value of is the pixel value of the input image, Is the convolution kernel at position The value of

[0042] Downsampling is done by maximum pooling and average pooling, where maximum pooling reduces the dimension of the feature map by selecting the maximum value in the pooling window. Using a 2×2 pooling window, the maximum pooling operation is expressed by the following formula:

[0043] (9)

[0044] Average pooling performs dimensionality reduction by selecting the average of all values ​​in the pooling window. Similar to maximum pooling, using a 2×2 pooling window, the average pooling operation is expressed as:

[0045] (10)

[0046] After completing the maximum pooling and average pooling, the output is concatenated using channel-wise merging. The concatenation is expressed as:

[0047] (11)

[0048] Convolution and pooling should be performed alternately to extract more and more abstract features layer by layer.

[0049] Optionally, the selection of difficult negative samples in the training process includes: for positive and negative sample pairs , first calculate the cosine similarity of the positive and negative sample pairs, the corresponding formula is:

[0050] (12)

[0051] in, and The samples and The eigenvector of represents the dot product, Represents the Euclidean norm, for positive samples , select the negative sample with the smallest distance from the negative sample set ,in is a set of negative sample pairs, and calculates their similarity:

[0052] (13)

[0053] Negative samples with high similarity are difficult negative samples;

[0054] NT-Xent loss is used to focus on distinguishing difficult negative samples. The loss function formula is:

[0055] (14)

[0056] in, is a positive sample, is a negative sample, is a hard negative sample, is the temperature coefficient.

[0057] Optionally, the network training is performed by adjusting the training process through the following steps, including:

[0058] Step S1: Select a pair of image samples from the training data and calculate their distance metric in the embedding space by cosine similarity;

[0059] Step S2: Based on the similarity or difference between paired samples, the model performance is evaluated by calculating the NT-Xent loss function. For each positive sample pair, their similarity is maximized, while for the negative sample pair, the similarity of all samples in the denominator is compared to ensure that the similarity of the negative sample is minimized.

[0060] Step S3, by adjusting the temperature parameters Scale the similarity, which in turn affects the calculation process of cosine similarity. Temperature parameter The smoothness of the loss function is controlled. A lower temperature will make the model focus more on the differences between similar sample pairs, while a higher temperature will increase the influence of negative sample pairs, resulting in a smoother learning process for the model.

[0061] Optionally, the score generation module predicts the image quality score based on the high-level feature vector extracted from the U-Net, using a fully connected regression network comprising an input layer, multiple fully connected layers, and an output layer, wherein the input layer is the high-level feature vector extracted by the U-Net, and its shape should be the same as the dimension of the feature vector;

[0062] Assume the fully connected input is , the output of this layer It can be calculated by the following formula:

[0063] (15)

[0064] in It is The weight matrix of the layer, It is The bias vector of the layer, yes Activation function, assuming that the output of the fully connected layer is , the calculation formula of the output layer is:

[0065] (16)

[0066] is the quality score of the image, is the weight matrix of the output layer, is the bias vector of the output layer.

[0067] Optionally, the training of the score generation module includes:

[0068] Step SS1, importing a dataset containing images and their corresponding true quality scores;

[0069] Step SS2: normalize all images and input them into the feature extraction module for feature extraction;

[0070] Step SS3, predicting the quality score of the image;

[0071] In step SS4, the mean square error (MSE) is used as the loss function to measure the difference between the model's predicted score and the actual score. The corresponding formula is:

[0072] (17)

[0073] in, It is a real subjective rating. is the score predicted by the model, and N is the size of the dataset;

[0074] Step SS5: Back propagate the error of each layer and calculate the gradient of each layer parameter ;

[0075] Step SS6, choose an appropriate learning rate , according to the calculated gradient, update the parameters according to the gradient descent rule, iterate multiple times until the loss function converges, and the update formula is: .

[0076] Compared with the prior art, the present invention has the following beneficial effects:

[0077] The present invention provides a method for generating training samples through data enhancement, extracting features using CNN, and evaluating image quality using contrastive learning. Through multi-level feature extraction and high-level feature calculation, the model's ability to judge image quality is enhanced. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 is a flow chart of a method according to an embodiment of the present invention;

[0079] Figure 2 This is a structural diagram of an image preprocessing module according to an embodiment of the present invention;

[0080] Figure 3 This is a structural diagram of the positive and negative sample pair construction and feature extraction module according to an embodiment of the present invention;

[0081] Figure 4 This is a diagram of a network training structure according to an embodiment of the present invention;

[0082] Figure 5 This is a structural diagram of a rating generation module according to an embodiment of the present invention;

[0083] Figure 6 This is a diagram of the fully connected regression network training structure of an embodiment of the present invention. DETAILED DESCRIPTION

[0084] In order to make the technical means, creative features, objectives and effects achieved by the present invention easier to understand, the present invention is further described below in conjunction with specific implementation methods.

[0085] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, features defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0086] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they may refer to fixed connections, detachable connections, or integral connections; mechanical connections or electrical connections; direct connections or indirect connections through an intermediate medium; and internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0087] like Figures 1-6 As shown, the present invention discloses an image quality assessment method based on data enhancement and contrastive learning, comprising:

[0088] Step 1: Preprocess the reference image and the distorted image;

[0089] Step 2: Input the preprocessed image into a pre-built image quality assessment model to obtain an image quality score;

[0090] Please see Figure 2 ,In one embodiment, the preprocessing includes random data augmentation, deep image prior, and bilinear interpolation;

[0091] In one embodiment, the random data augmentation process involves first performing a series of random data augmentation operations on the input image to generate more training samples. These augmentation operations include, but are not limited to, random flipping, random cropping, random rotation, random scaling and stretching, and color transformation. These augmentation operations are independently applied to each image in a probabilistic manner to generate more diverse training samples.

[0092] In one embodiment, in step 1, preprocessing the image includes: initializing an untrained convolutional neural network based on the depth image prior. The network architecture consists of a convolutional layer, an activation function, and a deconvolution layer. The initial parameters of the network are randomly initialized. The convolutional neural network (CNN) is used as a tool in the optimization process to directly repair the image without the need for any previous image dataset pre-training. The input of the network is a random noise image z, and the output is the repaired image. By minimizing the loss function, we optimize and adjust the network weights and biases so that the output image gradually becomes closer to the real image during the optimization process. First, we calculate the data loss. The operation is:

[0093] (1)

[0094] in, represents the pixel positions of all known regions, Represents the original image.

[0095] Then calculate the regularization loss to constrain the structure of the generated image and prevent overfitting or generating unreasonable images. The operation is:

[0096] (2)

[0097] in, is an image At pixel position The gradient indicates the degree of pixel change. For each pixel in the image To perform summation or calculation. Represents each pixel position in the image, and "all" refers to all pixels in the image.

[0098] The final loss function is the weighted sum of data loss and regularization loss, defined as:

[0099] +λ (3)

[0100] Where λ is a hyperparameter that controls the balance between data loss and regularization loss.

[0101] Finally, the gradient of the loss function with respect to each parameter (weight and bias) is calculated, and these gradients are backpropagated to update the network parameters. Backpropagation uses the chain rule to calculate the gradient of each parameter, passing it backward from the last layer to the input layer. The Adam optimizer dynamically adjusts the learning rate based on the gradient history of each parameter and performs parameter updates. This process is repeated until the loss function converges (reaches a minimum) or the maximum number of training cycles is reached.

[0102] In one embodiment, in step 1, the bilinear interpolation is used to perform secondary restoration on the image. First, the missing area in the depth image is identified, and the missing area is masked. To express, Indicates that the position is missing. Indicates that the position is known. For each missing point , select its left, right, top and bottom neighboring points for bilinear interpolation. The horizontal interpolation is expressed as:

[0103] (4)

[0104] (5)

[0105] is fixed, and Respectively expressed in and The linear interpolation result on .

[0106] Vertical interpolation can be expressed as:

[0107] (6)

[0108] The depth value of the missing area Update to the interpolated value;

[0109] Please see Figure 3 In one embodiment, in step 2, the positive and negative sample pairs are constructed using the output images of the three pre-processing steps, and a positive sample pair is constructed by making a one-to-one correspondence between the random data augmentation and the image generated by the bilinear interpolation method. The difference between the two is small, so there is a high similarity in the image content, which meets the characteristics of the positive sample. The images obtained by rotation, scaling and bilinear interpolation are first randomly shuffled. Since these images have large structural differences and pixel-level changes, the generated images have large visual and structural differences from the original images, so these transformed image pairs are marked as negative sample pairs.

[0110] In one embodiment, the positive and negative sample pairs are constructed, and after the positive and negative sample pairs are labeled, the image is normalized. The normalization uses scale normalization to scale all sample data to the same scale and symmetrically distribute them about 0. The specific normalization function is:

[0111] (7)

[0112] in, is the normalized image, is the input noise image.

[0113] In one embodiment, in step 2, the feature extraction module uses a U-Net encoder to convolve and downsample the input image to achieve the purpose of feature extraction; the hard negative mining technology is additionally used for negative samples to select "difficult" negative samples in the training process to promote the model to learn more discriminative feature representations.

[0114] In one embodiment, in step 2, the encoder includes a convolution layer and downsampling, and the convolution operation extracts local features of the image using a 3×3 convolution kernel. Assume that the input image is , then the convolution operation can be expressed as:

[0115] (8)

[0116] in Is the output feature map at position value. is the pixel value of the input image, Is the convolution kernel at position value.

[0117] Downsampling is done through maximum pooling and average pooling. Maximum pooling reduces the dimensionality of the feature map by selecting the maximum value in the pooling window. Using a 2×2 pooling window, the maximum pooling operation can be expressed as follows:

[0118] (9)

[0119] Average pooling performs dimensionality reduction by selecting the average of all values ​​in the pooling window. Similar to max pooling, using a 2×2 pooling window, the average pooling operation can be expressed as:

[0120] (10)

[0121] After completing the maximum pooling and average pooling, their outputs are concatenated using channel-wise merging so that the features of both pooling operations are retained and fed into the next layer. Their concatenation can be expressed as:

[0122] (11)

[0123] Convolution and pooling should be performed alternately to extract more and more abstract features layer by layer;

[0124] In one embodiment, in step 2, the difficult negative sample mining, for the positive and negative sample pairs , first calculate the cosine similarity of the positive and negative sample pairs, the corresponding formula is

[0125] (12)

[0126] in, and The samples and The eigenvector of represents the dot product, represents the Euclidean norm. For positive samples , select the negative sample with the smallest distance from the negative sample set ,in is a set of negative sample pairs), and calculate their similarity:

[0127] (13)

[0128] This negative sample with higher similarity is called a difficult negative sample.

[0129] In one embodiment, in step 2, the loss function calculation can be performed by adjusting the loss function so that it pays more attention to distinguishing these difficult negative samples, using NT-Xent loss; the loss function formula is:

[0130] (14)

[0131] in, is a positive sample, is a negative sample, is a hard negative sample, is the temperature coefficient.

[0132] Please see Figure 4 In one embodiment, in step 2, the network training is adjusted by the following steps, and the training process includes the following steps:

[0133] Step S1: Select a pair of image samples from the training data, which can be a positive sample pair or a negative sample pair, and calculate their distance metric in the embedding space by cosine similarity;

[0134] In step S2, the model's performance is evaluated by calculating the NT-Xent loss function based on the similarities or differences between paired samples. For each positive sample pair, their similarity is maximized. For negative sample pairs, the similarities of all samples in the denominator are compared to ensure that the similarity of the negative samples is minimized.

[0135] Step S3, temperature parameters Scale the similarity, which in turn affects the calculation process of cosine similarity. Temperature parameter Controls the smoothness of the loss function. A lower temperature will make the model focus more on the differences between similar pairs of samples, while a higher temperature will increase the influence of negative pairs, resulting in a smoother learning process for the model.

[0136] Please see Figure 5 In one embodiment, in step 2, the score generation module predicts the image quality score based on the high-level feature vector extracted from the U-Net. For this purpose, a fully connected regression network is used, comprising an input layer, multiple fully connected layers, and an output layer. The input layer is fed with the high-level feature vector extracted by the U-Net, and its shape should be the same as the dimension of the feature vector.

[0137] Assume the fully connected input is , the output of this layer It can be calculated by the following formula:

[0138] (15)

[0139] in It is The weight matrix of the layer, It is The bias vector of the layer, yes Activation function. Assume that the output of the fully connected layer is (where L is the number of the last layer), the output layer is calculated as:

[0140] (16)

[0141] is the quality score of the image, is the weight matrix of the output layer, is the bias vector of the output layer;

[0142] Please see Figure 6 In one embodiment, in step 2, the score generation module is trained by the following steps:

[0143] Step SS1, importing a dataset containing images and their corresponding true quality scores;

[0144] The image and image quality score dataset uses TID2013;

[0145] Step SS2: normalize all images and input them into the feature extraction module for feature extraction;

[0146] The feature extraction method is the same as that described in step 2 to ensure the consistency of the training environment and the application environment;

[0147] Step SS3, predicting the quality score of the image;

[0148] The rating prediction method is the same as the rating generation module described in step 2;

[0149] In step SS4, the mean square error (MSE) is used as the loss function to measure the difference between the model's predicted score and the actual score. The corresponding formula is:

[0150] (17)

[0151] in, It is a real subjective rating. is the score predicted by the model, and N is the size of the dataset;

[0152] Step SS5: Back propagate the error of each layer and calculate the gradient of each layer parameter ;

[0153] Step SS6, choose an appropriate learning rate , based on the calculated gradient, update the parameters according to the gradient descent rule. Iterate multiple times until the loss function converges. The update formula is:

[0154]

[0155] The specific implementation method of each unit is the same as that of each step and will not be described in detail in the present invention.

[0156] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. An image quality assessment method based on data enhancement and contrastive learning, characterized in that: include: Preprocess the image and input the preprocessed image into a pre-built image quality evaluation model to obtain the image quality score; The preprocessing includes performing random data augmentation, deep image prior, and bilinear interpolation operations on the image in sequence; the image quality assessment model includes a feature extraction module, a network training module, and a score generation module. The feature extraction module is used to extract high-level features from the constructed positive and negative sample pairs. The network training is performed using images and high-level features. The score generation module is used to predict the image quality score based on the high-level feature vector. The bilinear interpolation is used to perform secondary restoration on the image; The deep image prior is an untrained convolutional neural network after initialization. The network architecture consists of a convolutional layer, an activation function, and a deconvolution layer. The initial parameters of the network architecture are randomly initialized. The convolutional neural network is used as an optimization tool to directly repair the image. The input of the network architecture is a random noise image z, and the output is the repaired image. ; The construction of the positive and negative sample pairs includes: The preprocessed image is matched one-to-one with the image generated by the bilinear interpolation method by the random data augmentation to construct a positive sample pair; The images obtained by rotation, scaling and bilinear interpolation are first randomly shuffled, and the image pairs after the shuffled transformation are marked as negative sample pairs; wherein, after marking the positive and negative sample pairs, the images are normalized, and the normalization uses scale normalization to scale all sample data to the same scale and is symmetrically distributed about 0.

2. The image quality assessment method based on data enhancement and contrastive learning according to claim 1, characterized in that: The deep image prior is to initialize an untrained convolutional neural network. The network architecture consists of a convolutional layer, an activation function, and a deconvolution layer. The initial parameters of the network architecture are randomly initialized. The convolutional neural network is used as an optimization tool to directly repair the image. The input of the network architecture is a random noise image z, and the output is the repaired image. ; By minimizing the loss function, we optimize and adjust the weights and biases of the network so that the output image gradually becomes a real image during the optimization process. First, we calculate the data loss. The operation is: （1） in, represents the pixel positions of all known regions, represents the restored image, represents the original image; Then calculate the regularization loss to constrain the structure of the generated image. The operation is: (2) in, is an image At pixel position The gradient indicates the degree of pixel change. Indicates the sum or calculation of each pixel x in the image, x represents each pixel position in the image, and all refers to all pixels in the image; The final loss function is the weighted sum of data loss and regularization loss, defined as: +λ (3) Among them, λ is a hyperparameter that controls the balance between data loss and regularization loss; Calculate the gradient of the loss function with respect to each parameter, back-propagate the calculated gradient, and update the parameters of the network. The back-propagation calculates the gradient of each parameter by the chain rule, and passes it back from the last layer to the input layer until the loss function converges or the predetermined maximum number of training times is reached.

3. The image quality assessment method based on data enhancement and contrastive learning according to claim 1, characterized in that: The specific normalization function is: (7) in, is the normalized image, is the input noise image.

4. The image quality assessment method based on data enhancement and contrastive learning according to claim 1, wherein: The bilinear interpolation is used to perform secondary restoration on the image. First, the missing area in the depth image is identified and the missing area is masked. To express, Indicates that the pixel position corresponding to the coordinate (x, y) of the mask M is missing, The pixel position corresponding to the coordinate (x, y) of the mask M is known. For each missing point , select its left, right, top and bottom neighboring points for bilinear interpolation, where horizontal interpolation is expressed as: (4) (5) is fixed, and Respectively expressed in and The linear interpolation result on ; the vertical interpolation is expressed as: (6) The depth value of the missing area Updated to the interpolated value.

5. The image quality assessment method based on data enhancement and contrastive learning according to claim 1, characterized in that: The feature extraction module adopts a U-Net encoder, which uses the U-Net encoder to convolve and downsample the input image to achieve the purpose of feature extraction; a difficult negative sample mining method is additionally used for negative samples, and difficult negative samples in the training process are selected to promote the model to learn more discriminative feature representations.

6. The image quality assessment method based on data enhancement and contrastive learning according to claim 5, characterized in that: The U-Net encoder includes convolution layers and downsampling. The convolution operation extracts local features of the image. Using a 3×3 convolution kernel, assuming the input image is , then the convolution operation can be expressed as: (8) in Is the output feature map at position The value of is the pixel value of the input image, Is the convolution kernel at position The value of Downsampling is done by maximum pooling and average pooling, where maximum pooling reduces the dimension of the feature map by selecting the maximum value in the pooling window. Using a 2×2 pooling window, the maximum pooling operation is expressed by the following formula: (9) Average pooling performs dimensionality reduction by selecting the average of all values ​​in the pooling window. Similar to maximum pooling, using a 2×2 pooling window, the average pooling operation is expressed as: (10) After completing the maximum pooling and average pooling, the output is concatenated using channel-wise merging. The concatenation is expressed as: (11) Convolution and pooling should be performed alternately to extract more and more abstract features layer by layer.

7. The image quality assessment method based on data enhancement and contrastive learning according to claim 5, characterized in that: The difficult negative samples in the selection training process include: , first calculate the cosine similarity of the positive and negative sample pairs, the corresponding formula is: (12) in, and The samples and The eigenvector of represents the dot product, Represents the Euclidean norm, for positive samples , select the negative sample with the smallest distance from the negative sample set ,in is a set of negative sample pairs, and calculates their similarity: (13) Negative samples with high similarity are difficult negative samples; NT-Xent loss is used to focus on distinguishing difficult negative samples. The loss function formula is: (14) in, is a positive sample, is a negative sample, is a hard negative sample, is the temperature coefficient.

8. The image quality assessment method based on data enhancement and contrastive learning according to claim 1, characterized in that: The network training is performed by adjusting the training process through the following steps, including: Step S1: Select a pair of image samples from the training data and calculate their distance metric in the embedding space by cosine similarity; Step S2: Based on the similarity or difference between paired samples, the model performance is evaluated by calculating the NT-Xent loss function. For each positive sample pair, their similarity is maximized, while for the negative sample pair, the similarity of all samples in the denominator is compared to ensure that the similarity of the negative sample is minimized. Step S3, by adjusting the temperature parameters Scaling the similarity affects the calculation process of cosine similarity, temperature parameter The smoothness of the loss function is controlled. A lower temperature will make the model focus more on the differences between similar sample pairs, while a higher temperature will increase the influence of negative sample pairs, resulting in a smoother learning process for the model.

9. The image quality assessment method based on data enhancement and contrastive learning according to claim 1, wherein: The score generation module predicts the image quality score based on the high-level feature vector extracted from the U-Net. It uses a fully connected regression network, including an input layer, multiple fully connected layers, and an output layer. The input layer is the high-level feature vector extracted by the U-Net, and its shape should be the same as the dimension of the feature vector. Assume that the input of the fully connected layer is , No. The output of the fully connected layer Calculated by the following formula: (15) in It is The weight matrix of the layer, It is The bias vector of the layer, yes Activation function, assuming that the output of the fully connected layer is , the calculation formula of the output layer is: (16) is the quality score of the image, is the weight matrix of the output layer, is the bias vector of the output layer.

10. The image quality assessment method based on data enhancement and contrastive learning according to claim 1, characterized in that: The training of the score generation module includes: Step SS1, importing a dataset containing images and their corresponding true quality scores; Step SS2: normalize all images and input them into the feature extraction module for feature extraction; Step SS3, predicting the quality score of the image; In step SS4, the mean square error (MSE) is used as the loss function to measure the difference between the model's predicted score and the actual score. The corresponding formula is: (17) in, It is a real subjective rating. is the score predicted by the model, and N is the size of the dataset; Step SS5: Back propagate the error of each layer and calculate the gradient of each layer parameter ; Step SS6, choose an appropriate learning rate , according to the calculated gradient, update the parameters according to the gradient descent rule, iterate multiple times until the loss function converges, and the update formula is: .

Images