Image quality evaluation system and method based on domain self-adaption

Abstract

The invention discloses an image quality evaluation system and method based on domain self-adaption. The system comprises a data set construction module, a network construction module, a mutual supervision module, a parameter updating module and a quality evaluation module. The data set construction module is used for constructing a source domain data set and a target domain data set for image quality evaluation; the network construction module is used for constructing a domain self-adaptive twin network, and guiding a sub-network to pay attention to an easy-to-transfer region and a difficult-to-transfer region respectively based on domain discrimination results of a source domain data set and a target domain data set; the mutual supervision module screens high-confidence pseudo tags based on entropy sorting, so that sub-networks in the domain adaptive twin network mutually supervise prediction results of target domains of opposite parties through the pseudo tags; the parameter updating module updates parameters of the domain adaptive twin network based on global loss to obtain an image quality evaluation model; and the quality evaluation module performs image quality evaluation on the to-be-evaluated image by using the image quality evaluation model.

Description

Technical Field

[0001] The present invention belongs to the technical field of image quality evaluation, and in particular relates to an image quality evaluation system and method based on domain adaptation. Background Art

[0002] Deep learning algorithms have shown superior performance in the field of image quality assessment. However, when there is a large domain shift between the source domain training data and the target domain test data, the performance of deep learning-based image quality assessment methods tends to drop significantly. To solve this problem, domain-adaptive image quality assessment methods have emerged. They extract domain-invariant features by minimizing the maximum mean difference or adversarial training, thereby effectively alleviating the problem of performance degradation of deep learning-based image quality assessment methods in the case of domain shift.

[0003] The current domain-adaptive image quality assessment methods face the dilemma of balancing transferability and discriminability when extracting domain-invariant features. Image quality assessment is a complex task that requires comprehensive consideration of the degree of distortion and the main content. The discriminative features of an image usually come from the difficult-to-transfer area that reflects the main content. There is usually a large difference between the main content of the source domain and the target domain. Forcibly aligning the features of the difficult-to-transfer area can easily cause mode collapse and fail to achieve good domain adaptation effects. The easy-to-transfer features of an image usually come from the background area of ​​the image. Forcibly aligning the easy-to-transfer area features of the source domain and the target domain will lose content information, resulting in a decrease in discriminative performance. Summary of the invention

[0004] The present invention aims to solve the deficiencies of the prior art and provides the following solutions:

[0005] An image quality evaluation system based on domain adaptation includes: a data set construction module, a network construction module, a mutual supervision module, a parameter updating module and a quality evaluation module;

[0006] The data set construction module is used to construct a source domain data set and a target domain data set for image quality evaluation;

[0007] The network construction module is used to construct a domain-adaptive twin network, and guide the sub-network to focus on easy-to-transfer areas and difficult-to-transfer areas respectively based on the domain discrimination results of the source domain dataset and the target domain dataset;

[0008] The mutual supervision module selects high-confidence pseudo labels based on entropy sorting, so that the sub-networks in the domain adaptive twin network supervise each other's prediction results of the target domain through the pseudo labels;

[0009] The parameter updating module updates the parameters of the domain adaptive twin network based on the global loss to obtain an image quality evaluation model;

[0010] The quality assessment module uses the image quality assessment model to perform image quality assessment on the image to be assessed.

[0011] Preferably, the data set construction module includes: an initial data acquisition unit and a data processing unit;

[0012] The initial data acquisition unit is used to acquire image data, wherein the image data includes initial source domain data and initial target domain data;

[0013] The data processing unit is used to perform data enhancement on the initial source domain data and the initial target domain data to obtain the source domain data set and the target domain dataset in, represents the source domain image data, represents the target domain image data, Represents the image quality label of the source domain.

[0014] Preferably, the domain adaptive twin network is composed of a sub-network focusing on easy-to-transfer areas and a sub-network focusing on difficult-to-transfer areas;

[0015] The focus-on-transferable region subnetwork includes: a feature extractor Local area discriminator Global Domain Discriminator and score distribution classifier

[0016] The sub-network focusing on the difficult-to-transfer region includes: a feature extractor Local area discriminator Global Domain Discriminator and score distribution classifier

[0017] Preferably, the source domain label loss of the domain adaptive twin network is constructed by a feature extractor and a score distribution classifier:

[0018] Extracting a first global quality feature of the source domain image data by the feature extractor;

[0019] Mapping the first global quality feature into a first quality score distribution of source domain data by the score distribution classifier;

[0020] Aggregate the first quality score distribution into a mean opinion score, calculate a root mean square error based on the mean opinion score and the source domain quality label, and construct the source domain label loss, wherein the source domain label loss includes a sub-network source domain label loss focusing on an easy-to-transfer region and a sub-network source domain label loss focusing on a difficult-to-transfer region:

[0021]

[0022]

[0023] in, represents the source domain label loss of the sub-network focusing on the easily transferable region, n s Represents the total amount of source domain data, represents the mean opinion score of the subnetwork that focuses on the easily transferable region, Indicates the concern about the source domain label loss of the sub-network in the difficult-to-transfer region, Represents the mean opinion score of the sub-network focusing on the hard-to-transfer region.

[0024] Preferably, the global domain discriminator encourages the domain adaptive twin network to align global quality features by constructing a global domain discriminant loss;

[0025] The global domain discrimination loss includes: a global domain discrimination loss of a sub-network focusing on an easy-to-transfer region and a global domain discrimination loss of a sub-network focusing on a difficult-to-transfer region:

[0026]

[0027]

[0028] in, Indicates the global domain discrimination loss of the sub-network focusing on the easy-to-transfer region, Indicates the global domain discrimination loss of the sub-network focusing on the difficult-to-transfer region, n = n s +n t , n t represents the amount of data in the target domain, D = D s ∪D t , L ce represents the cross entropy loss, * represents the source domain / target domain, Indicates the field label.

[0029] Preferably, the local domain discriminator stimulates the domain adaptive twin network to align local quality features by constructing a local domain discrimination loss, and guides the two sub-networks in the domain adaptive twin network to focus on the easy-to-transfer area and the difficult-to-transfer area respectively through the entropy of the domain discrimination result of the local domain discriminator;

[0030] The local domain discrimination loss includes: focusing on the local domain discrimination loss of the easy-to-transfer region sub-network and focusing on the local domain discrimination loss of the difficult-to-transfer region sub-network:

[0031]

[0032]

[0033] in, Expressing concern about the local domain discrimination loss of the sub-network in the easy-to-transfer region, represents the local domain discrimination loss of the sub-network focusing on the difficult-to-transfer region, R represents the number of blocks in the image division region, Indicates the label of the area described by each area, Represents different areas of the image.

[0034] Preferably, the workflow of the mutual supervision module includes:

[0035] extracting a second global quality feature of the target domain image data by the feature extractor;

[0036] Mapping the second global quality feature into a second quality score distribution of target domain data by the score distribution classifier;

[0037] Calculate the entropy of the second quality score distribution, perform entropy sorting on the second quality score distribution of the target domain prediction of the domain adaptive twin network based on the obtained entropy value, set a screening ratio that increases with the number of training steps, and screen out a high-confidence target domain score distribution of the target domain prediction with the smallest entropy based on the screening ratio;

[0038] Aggregate the high-confidence target domain score distribution to obtain a high-confidence image quality pseudo label, wherein the high-confidence image quality pseudo label includes: a sub-network image quality pseudo label focusing on an easy-to-transfer region and a sub-network image quality pseudo label focusing on a difficult-to-transfer region:

[0039]

[0040]

[0041] in, Indicates the pseudo quality label of the image in the sub-network that focuses on the easily transferable region, represents the probability that the i-th confidence distribution score of the target domain predicted by the sub-network focusing on the easy-to-transfer region is j, Indicates the pseudo label of the image quality of the sub-network focusing on the difficult-to-transfer region, represents the probability that the i-th confidence distribution score of the target domain predicted by the sub-network focusing on the difficult-to-transfer region is j;

[0042] The sub-network focusing on the easy-to-transfer region and the sub-network focusing on the difficult-to-transfer region supervise the target domain prediction results of the other network respectively through the high-confidence image quality pseudo labels, and obtain the mutual supervision task loss of the sub-network focusing on the easy-to-transfer region and the mutual supervision task loss of the sub-network focusing on the difficult-to-transfer region:

[0043]

[0044]

[0045] in, Expressing concern about the loss of mutual supervision tasks in the sub-networks of the easily transferable region, Indicates the loss of mutual supervision task of sub-networks in the difficult-to-transfer region, n c represents the number of high-confidence image quality pseudo labels, Indicates the high-confidence image quality pseudo labels predicted by the sub-network that selects and focuses on the difficult-to-transfer region The mean opinion score of the corresponding data, Indicates the high-confidence image quality pseudo labels predicted by the sub-network that selects and focuses on the easily transferable regions The mean opinion score of the corresponding data.

[0046] Preferably, the workflow of the parameter updating module includes:

[0047] Self-supervised training loss is introduced, and the network global loss is constructed based on source domain label loss, global neighborhood discrimination loss, local neighborhood discrimination loss, self-supervised training loss and mutual supervision task loss:

[0048]

[0049]

[0050] Among them, L A represents the global loss of the sub-network focusing on the easily transferable region, L B represents the global loss of the sub-network focusing on the difficult-to-transfer region, represents the self-supervised loss of the sub-network focusing on the easily transferable region, represents the self-supervised loss of the sub-network focusing on the difficult-to-transfer region, and α, β, γ, λ, ω are hyperparameters for balancing the various losses;

[0051] The global loss is used to update the parameters of the domain adaptive twin network to obtain an image quality evaluation model.

[0052] The present invention also provides an image quality evaluation method based on domain adaptation, which is applied to any of the evaluation systems described above and comprises the following steps:

[0053] Construct source domain dataset and target domain dataset for image quality evaluation;

[0054] Constructing a domain-adaptive twin network, and guiding the sub-network to focus on easy-to-transfer areas and difficult-to-transfer areas respectively based on the domain discrimination results of the source domain dataset and the target domain dataset;

[0055] Screening high-confidence pseudo labels based on entropy sorting, so that the sub-networks in the domain adaptive twin network supervise each other's prediction results of the target domain through the pseudo labels;

[0056] The parameters of the domain adaptive twin network are updated based on the global loss to obtain an image quality evaluation model;

[0057] The image quality assessment model is used to perform image quality assessment on the image to be assessed.

[0058] Compared with the prior art, the present invention has the following beneficial effects:

[0059] The present invention solves the difficult problem of balancing transferability and discriminability in the process of domain adaptation of image quality assessment. By constructing a domain-adaptive twin network that focuses on easy-to-transfer areas and difficult-to-transfer areas, a high-confidence image quality pseudo-label is obtained through a course-based pseudo-label screening and warm-up module, and the target domain prediction results of the other network are supervised based on the high-confidence image quality pseudo-label, so as to encourage the twin network to obtain feature information that is both transferable and discriminative, and improve the performance of the domain-adaptive image quality assessment method. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0061] Figure 1 A schematic diagram of the system structure of an embodiment of the present invention;

[0062] Figure 2 This is a schematic diagram of the domain adaptive twin network structure of an embodiment of the present invention. DETAILED DESCRIPTION

[0063] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0064] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0065] Embodiment 1

[0066] In this embodiment, if Figure 1 As shown, an image quality assessment system based on domain adaptation includes: a data set construction module, a network construction module, a mutual supervision module, a parameter updating module and a quality assessment module.

[0067] The dataset construction module is used to construct source domain datasets and target domain datasets for image quality evaluation.

[0068] The dataset construction module includes: an initial data acquisition unit and a data processing unit; the initial data acquisition unit is used to acquire image data, and the image data includes initial source domain data and initial target domain data; the data processing unit is used to perform data enhancement on the initial source domain data and the initial target domain data to obtain the source domain dataset and target domain dataset in, represents the source domain image data, represents the target domain image data, Represents the image quality label of the source domain.

[0069] In this embodiment, data enhancement is performed on the image data of the source domain and the target domain, and the enhancement means include but are not limited to random rotation, random cropping, random mirroring, scaling, etc. In this embodiment, the image data of the source domain and the target domain are first scaled proportionally based on the short side, the short side is scaled to 384, and the long side is scaled according to the original ratio of the long and short sides, and then the image blocks of the size are randomly cropped, and finally random horizontal mirroring is used as data enhancement to form the source domain dataset and the target domain dataset for domain adaptive training.

[0070] The network construction module is used to construct a domain-adaptive twin network, and guides the sub-network to focus on easy-to-transfer areas and difficult-to-transfer areas based on the domain discrimination results of the source domain dataset and the target domain dataset, respectively.

[0071] In this embodiment, if Figure 2 As shown in Figure 1, the domain adaptive twin network consists of a sub-network focusing on easy-to-transfer regions and a sub-network focusing on difficult-to-transfer regions. The sub-network focusing on easy-to-transfer regions includes: feature extractor Local area discriminator Global Domain Discriminator and score distribution classifier The subnetwork focusing on the difficult-to-transfer area includes: Feature extractor Local area discriminator Global Domain Discriminator and score distribution classifier

[0072] The source domain label loss of the domain adaptive twin network is constructed through the feature extractor and the score distribution classifier: the first global quality feature of the source domain image data is extracted through the feature extractor; the first global quality feature is mapped to the first quality score distribution of the source domain data through the score distribution classifier; the first quality score distribution is aggregated into the mean opinion score, and the root mean square error is calculated based on the mean opinion score and the source domain quality label to construct the source domain label loss.

[0073] In this embodiment, for the sub-network focusing on the easy transfer area, the feature extractor Extract image data from the source domain Global quality characteristics Then through the score distribution classifier The global quality features extracted from the source domain Mapped to the quality score distribution of source domain data

[0074]

[0075] in, = represents the probability of the i-th image getting the j-score predicted by the sub-network focusing on the easily transferable region. Aggregate to mean opinion score

[0076]

[0077] Aggregated mean opinion score Source domain quality labels The root mean square error is calculated to construct the source domain label loss of the sub-network that focuses on the easy-to-transfer region, so as to learn the quality discrimination knowledge implicit in the source domain label data; similarly, the source domain label loss of the sub-network that focuses on the difficult-to-transfer region is constructed through the above steps:

[0078]

[0079]

[0080] in, represents the source domain label loss of the sub-network focusing on the easy-to-transfer region, n s Represents the total amount of source domain data, represents the mean opinion score of the subnetwork that focuses on the easily transferable region, Indicates the concern about the source domain label loss of the sub-network in the difficult-to-transfer region, Represents the mean opinion score of the sub-network focusing on the hard-to-transfer region.

[0081] The global domain discriminator encourages the domain adaptive twin network to align global quality features by constructing a global domain discrimination loss; the global domain discrimination loss includes: the global domain discrimination loss of the sub-network focusing on the easy-to-transfer region and the global domain discrimination loss of the sub-network focusing on the difficult-to-transfer region:

[0082]

[0083]

[0084] in, Indicates the global domain discrimination loss of the sub-network focusing on the easy-to-transfer region, Indicates the global domain discrimination loss of the sub-network focusing on the difficult-to-transfer region, n = n s +n t , n t represents the amount of data in the target domain, D = D s ∪D t , L ce represents the cross entropy loss, * represents the source domain / target domain, Indicates the field label.

[0085] The local domain discriminator encourages the domain adaptive twin network to align local quality features by constructing a local domain discrimination loss, and guides the two sub-networks in the domain adaptive twin network to focus on easy-to-transfer areas and difficult-to-transfer areas respectively through the entropy of the domain discrimination results of the local domain discriminator; the local domain discrimination loss includes: the local domain discrimination loss of the sub-network focusing on the easy-to-transfer area and the local domain discrimination loss of the sub-network focusing on the difficult-to-transfer area:

[0086]

[0087]

[0088] in, Expressing concern about the local domain discrimination loss of the sub-network in the easy-to-transfer region, represents the local domain discrimination loss of the sub-network focusing on the difficult-to-transfer region, R represents the number of blocks in the image division region, Indicates the label of each area. Represents different areas of the image. It represents the local quality features of different regions of the image extracted by the sub-network focusing on the easily transferable region, denoted as It indicates the binary classification results of the local domain discriminator of the subnetwork focusing on the easy transfer area for different regional features; the area with a classification probability of 0.5 in the binary classification result indicates the area that can confuse the local discriminator, that is, the easy transfer area. Let the entropy solution function be H, and the entropy of the classification result be The local quality features extracted from different regions are weighted to guide the sub-network that focuses on easy-to-transfer regions to focus on easy-to-transfer regions. It indicates the local quality features of different regions of the image extracted by the sub-network focusing on the difficult-to-transfer region, denoted as It indicates the discrimination result of the local domain discriminator of the sub-network of the difficult-to-transfer region for the two-class local quality features of different regions. The extracted local quality features are weighted to guide the sub-network that focuses on the difficult-to-transfer area to focus on the difficult-to-transfer area. The above-mentioned entropy weighting method based on the discrimination result of the local domain discriminator is used to construct a twin network that focuses on the easy-to-transfer area and the difficult-to-transfer area respectively. Before the local quality features are sent to the local domain discriminator, the maximum and minimum game of the feature extractor and the local domain discriminator are unified through the gradient reversal layer. In this embodiment, the entropy weighting of the discrimination result of the local domain discriminator is only applied to the attention module of the last encoder layer of the feature extractor transformer. The local quality features extracted from different patches of the image are weighted using the entropy of the discrimination result of the local domain discriminator through the Hadamard product method to guide the two sub-networks to focus on the easy-to-transfer area and the difficult-to-transfer area in the domain adaptation process respectively.

[0089] The mutual supervision module selects high-confidence pseudo-labels based on entropy sorting, so that the sub-networks in the domain adaptive twin network can supervise each other's prediction results of the target domain through pseudo-labels.

[0090] The workflow of the mutual supervision module includes: extracting the second global quality feature of the target domain image data through the feature extractor; mapping the second global quality feature to the second quality score distribution of the target domain data through the score distribution classifier; calculating the entropy of the second quality score distribution, and entropy sorting the second quality score distribution of the target domain prediction of the domain adaptive twin network based on the obtained entropy value, setting a screening ratio that increases with the number of training steps, and screening out the high-confidence target domain score distribution of the target domain prediction with the smallest entropy based on the screening ratio; aggregating the high-confidence target domain score distribution to obtain a high-confidence image quality pseudo-label.

[0091] In this embodiment, for the sub-network focusing on the easily transferable region, the image data of the target domain Through feature extractor Obtain global quality characteristics of target domain data Then through the score distribution classifier The global quality features extracted from the target domain Mapped to the score distribution of the target domain data

[0092]

[0093] in, Indicates that the subnetwork focusing on the easily transferable region predicts the probability that the i-th image will get the j-score; calculates the score distribution of the target domain data Entropy According to the low entropy assumption that the score distribution should satisfy, the score distribution predicted by the sub-network of the focus-easy transfer region is sorted by entropy, and the high-confidence target domain score distribution predicted by the sub-network of the focus-easy transfer region with the smallest entropy is selected with η as the ratio. Next, we will focus on the high-confidence target domain distribution predicted by the easy-to-transfer region subnetwork. Aggregate to mean opinion score to get high confidence image quality pseudo labels Similarly, according to the above steps, we can get a high-confidence image quality pseudo label

[0094]

[0095]

[0096] in, Indicates the pseudo quality label of the image in the sub-network that focuses on the easily transferable region, represents the probability that the i-th confidence distribution score of the target domain predicted by the sub-network focusing on the easy-to-transfer region is j, Indicates the pseudo label of the image quality of the sub-network focusing on the difficult-to-transfer region, Indicates the probability that the target domain i-th confidence distribution score predicted by the sub-network focusing on the difficult transfer area is j; by continuously increasing the screening ratio η, the process of human learning from easy to difficult is simulated. The increase in the ratio η can be continuously increased according to the performance of the sub-network on the source domain test set, or it can be continuously increased with the number of training steps. In this embodiment, the screening ratio η is continuously increased in a gradual manner by training the number of training steps. Assume that the number of preheating training steps is T warm , the total number of training steps T,T warm <T:

[0097]

[0098] Wherein, η0 represents the benchmark screening ratio, and t represents the current number of training steps. The subnetwork focusing on the easy-to-transfer region and the subnetwork focusing on the difficult-to-transfer region supervise the target domain prediction results of each other's network through high-confidence image quality pseudo-labels, and obtain the mutual supervision task loss of the subnetwork focusing on the easy-to-transfer region and the mutual supervision task loss of the subnetwork focusing on the difficult-to-transfer region; in this embodiment, the two subnetworks set a screening ratio that increases with the number of training steps, and supervise each other's target domain prediction results based on the high-confidence image quality pseudo-labels obtained by the screening ratio. The target domain score distribution predicted by the subnetwork focusing on the easy-to-transfer region is Aggregate to mean opinion score Filter out high-confidence image-quality pseudo labels predicted by the sub-network that focuses on the hard-to-transfer regions The mean opinion score of the corresponding data is recorded as The mutual supervision task loss of subnetwork A is constructed by the root mean square error:

[0099]

[0100] Similarly, we will focus on the target domain score distribution predicted by the sub-network in the difficult-to-transfer region. Aggregate to mean opinion score Filter out high-confidence image-quality pseudo labels predicted by the subnetwork that focuses on the easily transferable regions The mean opinion score of the corresponding data is recorded as The mutual supervision task loss of subnetwork B is constructed by the root mean square error:

[0101]

[0102] in, Expressing concern about the loss of mutual supervision tasks in the sub-networks of the easily transferable region, Indicates the loss of mutual supervision task of sub-networks in the difficult-to-transfer region, n c represents the number of high-confidence image quality pseudo labels, n c =ηn t .

[0103] The parameter update module updates the parameters of the domain adaptive Siamese network based on the global loss to obtain the image quality evaluation model.

[0104] In this embodiment, in order to further improve the performance of the network, a self-supervised loss in the target domain is introduced. The loss consists of three parts: the distribution entropy of a single image prediction is minimized to encourage the network to make more confident score distribution predictions; the entropy of the average value of different score distribution probabilities within a minibatch is maximized to encourage the network to make a wider range of predictions for the scores of different images to accelerate the convergence of the network; and the distance between the predicted score distribution and the truncated discrete normal distribution with the same expectation and variance is minimized to simulate the objective law that the crowd scoring conforms to the normal distribution.

[0105] The workflow of the parameter update module includes: introducing self-supervised training loss, constructing the network global loss based on source domain label loss, global neighborhood discrimination loss, local neighborhood discrimination loss, self-supervised training loss and mutual supervision task loss:

[0106]

[0107]

[0108] Among them, LA represents the global loss of the sub-network focusing on the easily transferable region, L B represents the global loss of the sub-network focusing on the difficult-to-transfer region, represents the self-supervised loss of the sub-network focusing on the easily transferable region, represents the self-supervised loss of the sub-network focusing on the difficult-to-transfer area, α, β, γ, λ, ω are hyperparameters for balancing the various losses. The global loss is used to update the parameters of the domain adaptive twin network to obtain the image quality evaluation model.

[0109] The quality assessment module uses the image quality assessment model to perform image quality assessment on the image to be evaluated.

[0110] After the domain adaptation is completed, the target domain image is passed through the domain adaptive twin network to output the quality judgment result of the target domain image. The results are aggregated into the mean opinion score and the actual quality label is used to calculate the SROCC and PLCC to verify the effectiveness of the scheme. In this embodiment, the mean opinion score obtained by aggregation is fitted with 5 parameters before calculating PLCC to ensure the accuracy of quality evaluation.

[0111]

[0112] in, Represents the average opinion score after aggregation, and β1, β2, β3, β4, and β5 are parameters to be fitted. In this embodiment, the feature extractor uses ViT-B16 pre-trained on ImageNet-21k, the structure of the score distribution classifier is (FC-LeakyReLu-FC-LeakyReLu-FC-Softmax), and the structures of the global domain discriminator and the local domain discriminator are both (FC-ReLu-Dropout-FC-ReLu-Dropout-FC-Sigmoid). In this embodiment, the SGD optimizer is used to optimize the network weights, the weight decay weightdecay is set to 0.0001, and the momentum momentum is set to 0.9. The learning rate of the global domain discriminator and the local domain discriminator is 0.002, the learning rate of the feature extractor is 0.005, and the learning rate of the score distribution classifier is set to 0.001. The total number of training steps is 2000, and the number of preheating training steps is 500. The learning rate of each module increases linearly from 0 to the above preset learning rate in the first 1000 training steps, and decreases to 0 with a cosine decay function in the remaining training steps. The baseline screening ratio η is 0.2, and the number of warm-up training steps is 500. The hyperparameters of each loss are α, β, γ, λ, and ω, respectively.

[0113] Embodiment 2

[0114] In this embodiment, a domain-adaptive image quality assessment method includes the following steps:

[0115] Construct source domain datasets and target domain datasets for image quality evaluation; construct a domain adaptive twin network, and guide the sub-networks to focus on easy-to-transfer areas and difficult-to-transfer areas respectively based on the domain discrimination results of the source domain datasets and the target domain datasets; screen high-confidence pseudo-labels based on entropy sorting, so that the sub-networks in the domain adaptive twin network can supervise each other's prediction results of the target domain through pseudo-labels; update the parameters of the domain adaptive twin network based on the global loss to obtain an image quality evaluation model; use the image quality evaluation model to evaluate the image quality of the image to be evaluated.

[0116] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A domain-adaptive image quality assessment system, characterized in that: include: Dataset construction module, network construction module, mutual supervision module, parameter update module and quality evaluation module; The data set construction module is used to construct a source domain data set and a target domain data set for image quality evaluation; The network construction module is used to construct a domain-adaptive twin network, and guide the sub-network to focus on easy-to-transfer areas and difficult-to-transfer areas respectively based on the domain discrimination results of the source domain dataset and the target domain dataset; The mutual supervision module selects high-confidence pseudo labels based on entropy sorting, so that the sub-networks in the domain adaptive twin network supervise each other's prediction results of the target domain through the pseudo labels; The parameter updating module updates the parameters of the domain adaptive twin network based on the global loss to obtain an image quality evaluation model; The quality assessment module uses the image quality assessment model to perform image quality assessment on the image to be assessed.

2. According to claim 1, the image quality assessment system based on domain adaptation is characterized in that: The data set construction module includes: an initial data acquisition unit and a data processing unit; The initial data acquisition unit is used to acquire image data, wherein the image data includes initial source domain data and initial target domain data; The data processing unit is used to perform data enhancement on the initial source domain data and the initial target domain data to obtain the source domain data set and the target domain dataset in, represents the source domain image data, represents the target domain image data, Represents the image quality label of the source domain.

3. The image quality assessment system based on domain adaptation according to claim 2, characterized in that: The domain adaptive twin network is composed of a sub-network focusing on easy-to-transfer areas and a sub-network focusing on difficult-to-transfer areas; The focus-on-transferable region subnetwork includes: a feature extractor Local area discriminator Global Domain Discriminator and score distribution classifier The sub-network focusing on the difficult-to-transfer region includes: a feature extractor Local area discriminator Global Domain Discriminator and score distribution classifier 4. The image quality assessment system based on domain adaptation according to claim 3, characterized in that: The source domain label loss of the domain adaptive Siamese network is constructed through the feature extractor and the score distribution classifier: Extracting a first global quality feature of the source domain image data by the feature extractor; Mapping the first global quality feature into a first quality score distribution of source domain data by the score distribution classifier; Aggregate the first quality score distribution into a mean opinion score, calculate a root mean square error based on the mean opinion score and the source domain quality label, and construct the source domain label loss, wherein the source domain label loss includes a sub-network source domain label loss focusing on an easy-to-transfer region and a sub-network source domain label loss focusing on a difficult-to-transfer region: in, represents the source domain label loss of the sub-network focusing on the easily transferable region, n s Represents the total amount of source domain data, represents the mean opinion score of the subnetwork that focuses on the easily transferable region, Indicates the concern about the source domain label loss of the sub-network in the difficult-to-transfer region, Represents the mean opinion score of the sub-network focusing on the hard-to-transfer region.

5. The domain-adaptive image quality assessment system according to claim 4, characterized in that: The global domain discriminator encourages the domain adaptive twin network to align global quality features by constructing a global domain discriminant loss; The global domain discrimination loss includes: a global domain discrimination loss of a sub-network focusing on an easy-to-transfer region and a global domain discrimination loss of a sub-network focusing on a difficult-to-transfer region: in, Indicates the global domain discrimination loss of the sub-network focusing on the easy-to-transfer region, Indicates the global domain discrimination loss of the sub-network focusing on the difficult-to-transfer region, n = n s +n t , n t represents the amount of data in the target domain, D = D s ∪D t , L ce represents the cross entropy loss, * represents the source domain / target domain, Indicates the field label.

6. The image quality assessment system based on domain adaptation according to claim 5, characterized in that: The local domain discriminator encourages the domain adaptive twin network to align local quality features by constructing a local domain discrimination loss, and guides the two sub-networks in the domain adaptive twin network to focus on the easy-to-transfer area and the difficult-to-transfer area respectively through the entropy of the domain discrimination result of the local domain discriminator; The local domain discrimination loss includes: focusing on the local domain discrimination loss of the easy-to-transfer region sub-network and focusing on the local domain discrimination loss of the difficult-to-transfer region sub-network: in, Expressing concern about the local domain discrimination loss of the sub-network in the easy-to-transfer region, represents the local domain discrimination loss of the sub-network focusing on the difficult-to-transfer region, R represents the number of blocks in the image division region, Indicates the label of the area described by each area, Represents different areas of the image.

7. The domain-adaptive image quality assessment system according to claim 6, characterized in that: The workflow of the mutual supervision module includes: extracting a second global quality feature of the target domain image data by the feature extractor; Mapping the second global quality feature into a second quality score distribution of target domain data by the score distribution classifier; Calculate the entropy of the second quality score distribution, perform entropy sorting on the second quality score distribution of the target domain prediction of the domain adaptive twin network based on the obtained entropy value, set a screening ratio that increases with the number of training steps, and screen out a high-confidence target domain score distribution of the target domain prediction with the smallest entropy based on the screening ratio; Aggregate the high-confidence target domain score distribution to obtain a high-confidence image quality pseudo label, wherein the high-confidence image quality pseudo label includes: a sub-network image quality pseudo label focusing on an easy-to-transfer region and a sub-network image quality pseudo label focusing on a difficult-to-transfer region: in, Indicates the pseudo quality label of the image in the sub-network that focuses on the easily transferable region, represents the probability that the i-th confidence distribution score of the target domain predicted by the sub-network focusing on the easy-to-transfer region is j, Indicates the pseudo label of the image quality of the sub-network focusing on the difficult-to-transfer region, represents the probability that the i-th confidence distribution score of the target domain predicted by the sub-network focusing on the difficult-to-transfer region is j; The sub-network focusing on the easy-to-transfer region and the sub-network focusing on the difficult-to-transfer region supervise the target domain prediction results of the other network respectively through the high-confidence image quality pseudo labels, and obtain the mutual supervision task loss of the sub-network focusing on the easy-to-transfer region and the mutual supervision task loss of the sub-network focusing on the difficult-to-transfer region: in, Expressing concern about the loss of mutual supervision tasks in the sub-networks of the easily transferable region, Indicates the loss of mutual supervision task of sub-networks in the difficult-to-transfer region, n c represents the number of high-confidence image quality pseudo labels, Indicates the high-confidence image quality pseudo labels predicted by the sub-network that selects and focuses on the difficult-to-transfer region The mean opinion score of the corresponding data, Indicates the high-confidence image quality pseudo labels predicted by the sub-network that selects and focuses on the easily transferable regions The mean opinion score of the corresponding data.

8. The domain-adaptive image quality assessment system according to claim 7, characterized in that: The workflow of the parameter updating module includes: Self-supervised training loss is introduced, and the network global loss is constructed based on source domain label loss, global neighborhood discrimination loss, local neighborhood discrimination loss, self-supervised training loss and mutual supervision task loss: Among them, L A represents the global loss of the sub-network focusing on the easily transferable region, L B represents the global loss of the sub-network focusing on the difficult-to-transfer region, represents the self-supervised loss of the sub-network focusing on the easily transferable region, represents the self-supervised loss of the sub-network focusing on the difficult-to-transfer region, and α, β, γ, λ, ω are hyperparameters for balancing the various losses; The global loss is used to update the parameters of the domain adaptive twin network to obtain an image quality evaluation model.

9. A domain-adaptive image quality assessment method, the assessment method being applied to the assessment system according to any one of claims 1 to 8, characterized in that: The following steps are involved: Construct source domain dataset and target domain dataset for image quality evaluation; Constructing a domain-adaptive twin network, and guiding the sub-network to focus on easy-to-transfer areas and difficult-to-transfer areas respectively based on the domain discrimination results of the source domain dataset and the target domain dataset; Screening high-confidence pseudo labels based on entropy sorting, so that the sub-networks in the domain adaptive twin network supervise each other's prediction results of the target domain through the pseudo labels; The parameters of the domain adaptive twin network are updated based on the global loss to obtain an image quality evaluation model; The image quality assessment model is used to perform image quality assessment on the image to be assessed.

Images