A domain-generalized image quality assessment method, program, device, and storage medium

Abstract

This invention belongs to the field of image quality assessment technology, specifically relating to a domain-generalized image quality assessment method, program, device, and storage medium. This invention learns image quality features from multiple datasets through a shared feature extractor and constructs a quality-sensitive feature space using a quality-sensitive triplet loss. Its unique multi-regressor structure effectively preserves the domain-specific quality discrimination knowledge. During the testing phase, this invention assigns higher weights to the regressor predictions corresponding to the training dataset most similar to the unknown target dataset based on the domain discriminator's discrimination results, effectively integrating the prediction results of each regressor and improving generalization performance. This invention effectively solves the problem that domain-generalized image quality assessment methods with mixed datasets cannot effectively integrate the quality discrimination knowledge of each dataset, thus improving the generalization of deep learning-based image quality assessment methods.

Description

Technical Field

[0001] This invention belongs to the field of image quality assessment technology, and specifically relates to a generalized image quality assessment method, program, device and storage medium. Background Technology

[0002] Deep learning methods have demonstrated superior performance in image quality assessment. However, the performance of deep learning-based image quality assessment methods deteriorates significantly when there is a large distributional difference between the training and test data. Therefore, improving the generalization ability of deep learning models is crucial for enhancing the practicality of deep learning-based image quality assessment methods.

[0003] Mixing multiple image quality assessment datasets to augment data distribution is an effective way to improve the generalization of deep learning-based image quality assessment models. However, different image quality assessment datasets often have significant distributional differences due to variations in image distortion types, image content, and the different emotional biases of annotators. Simple dataset mixing may degrade model performance due to these large data distribution differences and fail to effectively preserve the domain-specific quality judgment knowledge of different datasets. Summary of the Invention

[0004] The purpose of this invention is to provide a method, program, device, and storage medium for image quality evaluation that is applicable to a wide range of fields.

[0005] A domain-generalized image quality assessment method includes the following steps:

[0006] Step 1: Obtain M datasets with image quality score labels, set a domain label for each dataset, and construct the M datasets with domain labels as the training set;

[0007] Step 2: Train a domain-generalized image quality assessment model using the training set;

[0008] The domain-generalized image quality assessment model includes a shared feature extractor, a regressor, and a domain discriminator; the number of regressors is M, corresponding to M datasets.

[0009] When training a domain-generalized image quality assessment model, the quality-sensitive triplet loss, quality prediction loss, monotonicity loss, and domain discrimination loss are calculated, and a global loss is constructed by weighting them.

[0010] Step 3: Input the image data sample to be evaluated into the trained domain generalized image quality evaluation model. After the shared feature extractor extracts the image data quality features, the image data quality features are input into the domain discriminator and M regressors respectively. The results of the domain discriminator are used as weights to weight and fuse the quality scores output by the M regressors to obtain the quality score of the image data sample to be evaluated, thus completing the image quality evaluation.

[0011] Furthermore, in step 1, the training set D contains M datasets, D = {D1, D2, ..., D...} k ,...,D M}, the k-th dataset D k The CPC has N k One sample, and Let d be the image data and quality score of the i-th sample in the k-th dataset. k Let be the domain label for the k-th dataset.

[0012] Furthermore, in step 2, the training phase dataset D k Image data of the i-th sample The input is fed into a shared feature extractor, which extracts image data quality features. With dataset D k The corresponding regressor R k Image data quality characteristics Mapped to quality score Domain Discriminator D c Based on image data quality characteristics Predict the domain label of the sample to obtain the sample Probability set corresponding to labels in each domain Let d represent the predicted domain label for the i-th sample. k The probability of.

[0013] Furthermore, the method for calculating the quality-sensitive triplet loss in step 2 is as follows:

[0014] M datasets are input into a domain-generalized image quality assessment model. For each sample... Use its own image data As anchor images, the superscript "*" indicates that the dataset does not distinguish the source of the samples; samples are randomly selected from M datasets. And satisfy ε is a preset quality difference threshold. Positive samples are used as anchor point images; samples are randomly selected from M datasets. And satisfy Will As a negative sample of the anchor image;

[0015] To shorten the distance between the anchor image and positive samples and increase the distance between the anchor image and negative samples in the feature space, the loss of the quality-sensitive triplet is calculated as follows:

[0016]

[0017] Where N is the total number of samples in the M datasets; δ is the minimum distance difference between the preset positive and negative samples and the anchor point image in the feature space.

[0018] Furthermore, the method for calculating the quality prediction loss in step 2 is as follows:

[0019]

[0020] in, Indicates the root mean square error;

[0021] The method for calculating the monotonicity loss is as follows:

[0022]

[0023] in, Represents dataset D k The corresponding regressor R k The monotonic loss, Err (i,j),k For dataset D k middle and The monotonicity error,

[0024] The method for calculating the domain discrimination loss is as follows:

[0025]

[0026] in, Represents cross-entropy loss;

[0027] The global loss L is constructed by weighting the quality-sensitive triplet loss, quality prediction loss, monotonicity loss, and domain discrimination loss as follows:

[0028] L=αL pre +βL mono +γL triplet +λL dom

[0029] Where α, β, γ, and λ are the weighting hyperparameters that balance the various losses.

[0030] Furthermore, in step 3, the image data sample x to be evaluated t The data is fed into a pre-trained, domain-generalized image quality assessment model, and a shared feature extractor extracts image data quality features G(x). t After that, the image data quality features G(x) are... t The results are input to the neighborhood discriminator and M regressors, respectively, and the output of the neighborhood discriminator is used as the basis for the calculation. The quality scores output by the M regressors are used as weights. Weighted fusion yields the quality score of the image data sample to be evaluated.

[0031]

[0032] Furthermore, after constructing the training set in step 1, data augmentation is performed on the image data of the training dataset. Augmentation methods include random rotation, random cropping, random mirroring, and scaling. In step 2, the shared feature extractor uses a feature extraction layer of ResNet18 pre-trained on ImageNet superimposed with two fully connected layers with ReLU as the activation function, denoted as FC-ReLU-FC-ReLU. The regressor has a structure of two linear layers, denoted as FC-FC. The neighborhood discriminator has a structure denoted as FC-ReLU-FC-ReLU-FC-SoftMax.

[0033] A computer device / apparatus / system includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the image quality assessment method generalized from the above-mentioned field.

[0034] A computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the steps of an image quality assessment method generalized from the above-mentioned field.

[0035] A computer program product includes a computer program / instructions that, when executed by a processor, implement steps of an image quality assessment method generalized from the above-described domain.

[0036] The beneficial effects of this invention are as follows:

[0037] This invention learns image quality features from multiple datasets through a shared feature extractor and constructs a quality-sensitive feature space using a quality-sensitive triplet loss. Its unique multi-regressor structure effectively preserves domain-specific quality discrimination knowledge. During testing, the invention assigns higher weights to the regressor predictions corresponding to the training dataset most similar to the unknown target dataset based on the domain discriminator's discrimination results, effectively integrating the prediction results of each regressor and improving generalization performance. This invention effectively solves the problem that domain-generalized image quality assessment methods with mixed datasets cannot effectively integrate the quality discrimination knowledge of each dataset, thus improving the generalization of deep learning-based image quality assessment methods. Attached Figure Description

[0038] Figure 1 This is a flowchart of a domain-generalized image quality evaluation method according to the present invention.

[0039] Figure 2 This is a schematic diagram of a domain-generalized image quality evaluation method provided in an embodiment of the present invention.

[0040] Figure 3 This is a schematic diagram of a quality-sensitive triplet provided for an embodiment of the present invention. Detailed Implementation

[0041] The present invention will now be further described with reference to the accompanying drawings.

[0042] This invention designs a domain-generalized image quality assessment model with a shared feature extractor and multiple regressor branches. It effectively learns quality discrimination knowledge of mixed datasets through quality-sensitive triplet loss, as well as quality prediction loss and monotonicity loss of each regressor branch. When facing an unknown target dataset, the domain discriminator trained with the domain discrimination loss will give higher weight to the regressor prediction results corresponding to the most similar training dataset, effectively integrating the prediction results of each regressor and improving the generalization performance of the model.

[0043] First, obtain M datasets with image quality score labels. For each dataset, set a domain label and construct the M datasets with domain labels as the training set.

[0044] We construct a training dataset D = {D1, D2, ..., D...} for a domain-generalized image quality assessment method by mixing M image quality assessment datasets with known quality scores. k ,...,D M}, where the k-th dataset is and Let d be the image data and quality score of the i-th sample in the k-th dataset. k Let N be the domain label of the k-th dataset.k For the k-th dataset D k The number of samples in the sample.

[0045] Before training, image data in the training dataset can be augmented. Augmentation methods include, but are not limited to, random rotation, random cropping, random mirroring, scaling, and other operations.

[0046] In one embodiment of the present invention, the image data is scaled proportionally based on the shorter side, which is scaled to 512 pixels, while the longer side is scaled according to the original image ratio. 512×512 pixel image blocks are randomly cropped and then randomly horizontally mirrored to form training data for a domain generalization model. In this embodiment, a total of four datasets are mixed.

[0047] A domain-generalized image quality assessment model is constructed, which consists of a shared feature extractor, M regressors, and a domain discriminator; the M regressors correspond one-to-one with the M datasets.

[0048] A shared feature extractor is used to learn the quality discrimination features inherent in each dataset. The regressors corresponding to each dataset that make up the training dataset are used to preserve the quality discrimination knowledge specific to each dataset. A domain discriminator is used to weight and aggregate the prediction results of the regressors corresponding to each dataset based on the domain discrimination results when testing on an unknown target dataset, thereby improving the model's generalization ability.

[0049] In one embodiment of the present invention, the feature extractor uses a ResNet18 feature extraction layer pre-trained on ImageNet, superimposed with two fully connected layers (FC-ReLu-FC-ReLu) with ReLU as the activation function. The structure of each regressor branch is two linear layers (FC-FC), and the structure of the neighborhood discriminator is (FC-ReLu-FC-ReLu-FC-SoftMax).

[0050] During the training phase, dataset D k Image data of the i-th sample The input is fed into a shared feature extractor, which extracts image data quality features. With dataset D k The corresponding regressor R k Image data quality characteristics Mapped to quality score Domain Discriminator D c Based on image data quality characteristics Predict the domain label of the sample to obtain the sample Probability set corresponding to labels in each domain Let d represent the predicted domain label for the i-th sample. k The probability of.

[0051] We construct quality-sensitive triplet loss, quality prediction loss, monotonicity loss, and domain discrimination loss, and train a domain-generalized image quality assessment model by combining all losses.

[0052] First, quality-discriminating triples are selected based on differences in quality scores, and quality-sensitive triple loss is constructed by extracting quality features through a shared feature extractor.

[0053] M datasets are input into a domain-generalized image quality assessment model, and image samples from the mixed dataset are selected based on quality score differences. The quality discrimination triplet is selected in the middle, with * indicating that the dataset source is not distinguished. A quality difference threshold ε is set, and image samples are selected during training. as anchor point image Samples with a quality score difference less than or equal to ε are considered positive samples. Samples with a quality score difference greater than ε are considered negative samples.

[0054] Extracting quality discrimination triples using a shared feature extractor The quality characteristics are analyzed and a quality-sensitive triplet loss is constructed, which brings the anchor point closer to the positive sample and widens the distance between the anchor point and the negative sample in the feature space.

[0055] The loss of the quality-sensitive triplet is:

[0056]

[0057] Where N is the total number of samples in the M datasets; δ is the minimum distance difference between the preset positive and negative samples and the anchor point image in the feature space.

[0058] In one embodiment of the present invention, the quality difference threshold ε is set to 0.1 and δ is set to 1.

[0059] Secondly, the regressors corresponding to each dataset that makes up the training dataset map the quality features to quality scores and compare them with the quality score labels to construct quality prediction loss and monotonicity loss;

[0060]

[0061] in, Indicates the root mean square error;

[0062] To ensure consistency between the quality scores obtained from the regressor mapping for each dataset and the ranking of the quality score labels, the monotonicity loss for the regressor corresponding to each dataset is constructed as follows:

[0063]

[0064] in, Represents dataset D k The corresponding regressor R k The monotonic loss; Err (i,j),k For dataset D k middle and Monotonicity error:

[0065]

[0066] in, and During training, these represent input image sample pairs respectively. The quality score labels and the model-predicted quality scores. sign(·) is the sign function. The total monotonicity loss for M regression branches is:

[0067]

[0068] The domain discriminator predicts the dataset to which the extracted quality features belong, and compares the prediction results with the domain labels to construct the domain discrimination loss:

[0069]

[0070] in, Represents cross-entropy loss;

[0071] By combining all losses, a global loss is constructed to train a domain-generalized image quality assessment model:

[0072] L=αL pre +βL mono +γL triplet +λL dom

[0073] Where α, β, γ, and λ are the weighting hyperparameters that balance the various losses.

[0074] Based on global loss, the image quality evaluation model is iteratively optimized to generalize to the domain.

[0075] In one embodiment of the present invention, α, β, γ, and λ are set to 1, 0.3, 0.1, and 1, respectively. The SGD optimizer is used to optimize the model weights, with weight decay set to 0.0001 and momentum set to 0.9. The learning rate is set to 0.005. The total number of training steps is 9000. The learning rate increases linearly from 0 to the preset learning rate in the first 1000 training steps, and then decays using a cosine decay function in the remaining training steps.

[0076] After training, the image data samples x to be evaluated are... tThe data is fed into a pre-trained, domain-generalized image quality assessment model, and a shared feature extractor extracts image data quality features G(x). t After that, the image data quality features G(x) are... t The results are input to the neighborhood discriminator and M regressors, respectively, and the output of the neighborhood discriminator is used as the basis for the calculation. The quality scores output by the M regressors are used as weights. Weighted fusion yields the quality score of the image data sample to be evaluated.

[0077]

[0078] In one embodiment of the present invention, the Spearman Rank Order Correlation Coefficient (SROCC) and Pearson Linear Correlation Coefficient (PLCC) are calculated between the final predicted score and the actual quality score label to verify the effectiveness of the scheme.

[0079] This invention proposes a domain-generalized image quality assessment method based on multiple datasets. It learns image quality features from multiple datasets through a shared feature extractor and constructs a quality-sensitive feature space using a quality-sensitive triplet loss. The unique multi-regressor structure effectively preserves the domain-specific quality discrimination knowledge. During the testing phase, the discrimination results of the domain discriminator assign higher weights to the regressor predictions corresponding to the training dataset most similar to the unknown target dataset, effectively integrating the prediction results of each regressor and improving generalization performance. This method effectively solves the problem that domain-generalized image quality assessment methods based on multiple datasets cannot effectively integrate the quality discrimination knowledge of each dataset, and effectively improves the generalization of deep learning-based image quality assessment methods.

[0080] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A domain-generalized image quality assessment method, characterized in that, Includes the following steps: Step 1: Obtain A dataset with image quality score labels is provided. For each dataset, a domain label is set, and the domain-labeled datasets are... Each dataset is constructed as a training set; Step 2: Train a domain-generalized image quality assessment model using the training set; The domain-generalized image quality assessment model includes a shared feature extractor, a regressor, and a domain discriminator; the number of regressors is [number missing]. One, with Each dataset corresponds to one dataset; When training a domain-generalized image quality assessment model, the quality-sensitive triplet loss, quality prediction loss, monotonicity loss, and domain discrimination loss are calculated, and a global loss is constructed by weighting them. The method for calculating the loss of the quality-sensitive triplet is as follows: Each dataset is input into a domain-generalized image quality assessment model, and for each sample... , to transfer its own image data As anchor images, the superscript "*" indicates that the dataset does not distinguish the source of the samples; from Randomly select samples from the dataset And satisfy , For the preset quality difference threshold, As positive samples of the anchor point image; from Randomly select samples from the dataset And satisfy ,Will As a negative sample of the anchor image; To shorten the distance between the anchor image and positive samples and increase the distance between the anchor image and negative samples in the feature space, the loss of the quality-sensitive triplet is calculated as follows: in, for The total number of samples in a dataset; This is the minimum distance difference between the preset positive and negative samples and the anchor point image in the feature space; Step 3: Input the image data samples to be evaluated into the trained domain-generalized image quality assessment model. After the shared feature extractor extracts the image data quality features, the image data quality features are input into the domain discriminator and... Each regressor uses the output of the domain discriminator as the weight pair. The quality scores output by each regressor are weighted and fused to obtain the quality score of the image data sample to be evaluated, thus completing the image quality evaluation.

2. The domain-generalized image quality assessment method according to claim 1, characterized in that: The training set in step 1 The CCP Data sets, , No. Datasets The CCP One sample, , and The first The first dataset Image data and quality scores for each sample. For the first Domain labels for each dataset.

3. The domain-generalized image quality assessment method according to claim 2, characterized in that: The training phase dataset in step 2 Middle Image data of one sample The input is fed into a shared feature extractor, which extracts image data quality features. , with dataset Corresponding regressor Image data quality characteristics Mapped to quality score Domain discriminator Based on image data quality characteristics Predict the domain label of the sample to obtain the sample Probability set corresponding to labels in each domain , Indicates the prediction of the first The domain labels for each sample are The probability of.

4. The domain-generalized image quality assessment method according to claim 3, characterized in that: The method for calculating the quality prediction loss in step 2 is as follows: in, Indicates the root mean square error; The method for calculating the monotonicity loss is as follows: in, Represents the dataset Corresponding regressor The monotonic loss, , For dataset middle and The monotonicity error, ; The method for calculating the domain discrimination loss is as follows: in, Represents cross-entropy loss; The global loss is constructed by weighting the quality-sensitive triplet loss, quality prediction loss, monotonicity loss, and domain discrimination loss. for: in, , , , The weighting hyperparameters are used to balance the various losses.

5. The domain-generalized image quality assessment method according to claim 3, characterized in that: In step 3, the image data samples to be evaluated are... The data is fed into a pre-trained, domain-generalized image quality assessment model, and a shared feature extractor extracts image data quality features. Then, the image data quality features The inputs are respectively fed into the neighborhood discriminator and Each regressor, with the result of the domain discriminator output. As weight pairs The quality score output by the regressor Weighted fusion yields the quality score of the image data sample to be evaluated. : 。 6. The domain-generalized image quality assessment method according to claim 1, characterized in that: After constructing the training set in step 1, data augmentation is performed on the image data of the training dataset. Augmentation methods include random rotation, random cropping, random mirroring, and scaling. In step 2, the shared feature extractor uses a feature extraction layer of ResNet18 pre-trained on ImageNet superimposed with two fully connected layers with ReLU activation function, represented as FC-ReLU-FC-ReLU. The regressor has a structure of two linear layers, represented as FC-FC. The neighborhood discriminator has a structure represented as FC-ReLU-FC-ReLU-FC-SoftMax.

7. A computer device / equipment / system, comprising a memory, a processor, and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 6.

8. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that: When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1 to 6.

9. A computer program product, comprising a computer program / instructions, characterized in that: When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1 to 6.

Images