Image quality assessment method, device, electronic device and machine-readable storage medium

Abstract

The present application provides an image quality assessment method, apparatus, electronic device, and machine-readable storage medium. The image quality assessment method comprises: extracting features from an input image using a feature extractor in a trained multi-attribute quality assessment model to obtain image features; and predicting scores for the image features using a regressor in the trained multi-attribute quality assessment model to obtain assessment scores for multiple different distortion types of the input image. The multi-attribute quality assessment model is trained using a training set that includes real distorted samples and synthetic distorted samples using feature alignment training. This method can improve model training efficiency and reduce the difficulty of sample preparation.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image quality assessment method, device, electronic device, and machine-readable storage medium. Background Art

[0002] In real-world scenarios, images can exhibit significant variations in image quality due to various external factors (such as occlusion, interference, and lighting variations) as well as image acquisition constraints (such as camera resolution and camera angle). Low-quality images not only consume significant storage resources but can also introduce erroneous information, resulting in unnecessary data loss. Therefore, image quality assessment has become an increasingly important research area. Summary of the Invention

[0003] In view of this, the present application provides an image quality assessment method, apparatus, electronic device, and machine-readable storage medium to improve the efficiency of image quality assessment.

[0004] Specifically, this application is implemented through the following technical solutions:

[0005] According to a first aspect of an embodiment of the present application, there is provided an image quality assessment method, comprising:

[0006] The feature extractor in the trained multi-attribute quality assessment model is used to extract features from the input image to obtain image features;

[0007] Using the regressor in the trained multi-attribute quality assessment model to perform score prediction on the image features, thereby obtaining assessment scores for multiple different distortion types of the input image;

[0008] The multi-attribute quality assessment model is trained using a feature alignment training method with a training set including real distorted samples and synthetic distorted samples.

[0009] According to a second aspect of an embodiment of the present application, there is provided an image quality assessment apparatus, comprising:

[0010] A feature extraction unit is used to extract features from the input image using the feature extractor in the trained multi-attribute quality assessment model to obtain image features;

[0011] a quality assessment unit, configured to perform score prediction on the image features using the regressor in the trained multi-attribute quality assessment model to obtain assessment scores for a plurality of different distortion types of the input image;

[0012] The multi-attribute quality assessment model is trained using a feature alignment training method with a training set including real distorted samples and synthetic distorted samples.

[0013] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising a processor and a memory, wherein the memory stores machine-executable instructions that can be executed by the processor, and the processor is used to execute the machine-executable instructions to implement the image quality assessment method provided in the first aspect.

[0014] According to a fourth aspect of an embodiment of the present application, a machine-readable storage medium is provided, wherein the machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed by a processor, the image quality assessment method provided in the first aspect is implemented.

[0015] The technical solution provided by this application can at least bring the following beneficial effects:

[0016] By using a training set that includes real distorted samples and synthetic distorted samples and a feature alignment training method to train the multi-attribute quality assessment model, the performance of the model can be guaranteed even when the number of manually labeled real distorted samples is small, the training efficiency of the model can be improved, and the difficulty of sample preparation can be reduced. In addition, when using the multi-attribute quality assessment model to assess the quality of images, it is no longer limited to evaluating the overall quality of the image, but can determine the specific distortion type of the image and perform quantitative evaluation of each distortion type of the image. Finally, the training and testing processes of the model are both end-to-end networks. After inputting the image, the score of each distortion type can be directly output, which is simple and efficient. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a flowchart of an image quality assessment method shown in an exemplary embodiment of the present application;

[0018] Figure 2 This is a schematic diagram of an overall process of image quality assessment shown in an exemplary embodiment of the present application;

[0019] Figure 3 is a structural diagram of a multi-attribute quality assessment model shown in an exemplary embodiment of the present application;

[0020] Figure 4 is a schematic diagram of a feature alignment training process shown in an exemplary embodiment of the present application;

[0021] Figure 5 is a structural diagram of an image quality assessment device shown in an exemplary embodiment of the present application;

[0022] Figure 6 is a structural diagram of another image quality assessment device shown in an exemplary embodiment of the present application;

[0023] Figure 7It is a schematic diagram of the hardware structure of an electronic device shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0024] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0025] The terms used in this application are for the purpose of describing particular embodiments only and are not intended to limit this application. As used in this application and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0026] In order to enable those skilled in the art to better understand the technical solutions provided by the embodiments of the present application, and to make the above-mentioned purposes, features and advantages of the embodiments of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application are further described in detail below with reference to the accompanying drawings.

[0027] See Figure 1 , is a flow chart of an image quality assessment method provided in an embodiment of the present application, such as Figure 1 As shown, the image quality assessment method may include the following steps:

[0028] Step S100: Using the feature extractor in the trained multi-attribute quality assessment model to extract features from the input image to obtain image features.

[0029] Step S110: Use the regressor in the trained multi-attribute quality assessment model to perform score prediction on the image features to obtain assessment scores of multiple different distortion types of the input image.

[0030] In an embodiment of the present application, in order to achieve quantitative evaluation of image quality, when performing image quality evaluation, it is no longer limited to evaluating the overall quality of the image, but it is possible to determine the specific distortion type of the image and perform quantitative evaluation of each distortion type of the image.

[0031] Accordingly, for any input image, the feature extractor in the trained multi-attribute quality assessment model can be used to extract features of the input image to obtain image features, and the regressor in the trained multi-attribute quality assessment model can be used to perform score prediction on the extracted image features to obtain evaluation scores of various different distortion types of the input image.

[0032] For example, considering that real distorted images require manual collection and annotation, which is inefficient and difficult to obtain, using a large number of real distorted images for model training results in low training efficiency. Furthermore, if the sample data used for model training is too small, the model's performance cannot be guaranteed.

[0033] Based on the above considerations, in an embodiment of the present application, in order to improve the efficiency of model training while ensuring model performance, when training the multi-attribute quality assessment model, the training set may include real distorted samples and synthetic distorted samples, and the multi-attribute quality assessment model may be trained based on the real distorted samples and the synthetic distorted samples using feature alignment training.

[0034] In one example, a synthetic distortion sample can be generated by adding interference to a high-quality image, and the label score of each distortion type of the synthetic distortion sample is determined by the strength of the enhanced interference, and the label score of each distortion type is negatively correlated with the strength of the interference added to the corresponding distortion type.

[0035] Exemplarily, the label scores of the above samples refer to the scores corresponding to each distortion type.

[0036] Among them, a high-quality image is an image whose evaluation score of each distortion type exceeds a preset threshold.

[0037] For example, a synthetic distorted image can be obtained by adding interference to a high-quality image, and the degree of the added interference is mapped to a label score corresponding to the distortion attribute.

[0038] Exemplary ways of adding interference to high-quality images may include, but are not limited to, one or more of the following:

[0039] Adjust brightness, compress the image, and add noise.

[0040] Exemplarily, a high-quality image is an image whose evaluation score of each distortion type exceeds a preset threshold, and the image can be selected manually or with the assistance of an existing scoring model.

[0041] In an embodiment of the present application, during the model training process, the common features between the real distorted samples and the synthetic distorted samples are determined, and based on the common features between the real distorted samples and the synthetic distorted samples, the model training is performed using a feature alignment training method. Thus, even when the number of real distorted samples is small (for example, the proportion of real distorted samples to the total number of samples in the training set is less than a preset proportion threshold), the model's processing performance for real distorted samples can still be guaranteed.

[0042] It can be seen that in Figure 1In the method flow shown, the multi-attribute quality assessment model is trained by utilizing a training set including real distorted samples and synthetic distorted samples using feature alignment training. Thus, the performance of the model can be guaranteed even when the number of manually labeled real distorted samples is small, thereby improving the training efficiency of the model and reducing the difficulty of sample preparation. In addition, when using the multi-attribute quality assessment model to assess the quality of an image, it is no longer limited to assessing the overall quality of the image, but can determine the specific distortion type of the image and perform quantitative assessment of each distortion type of the image. Finally, the training and testing processes of the model are both end-to-end networks. After inputting an image, the score of each distortion type can be directly output, which is concise and efficient.

[0043] In some embodiments, the multi-attribute quality assessment model is trained by:

[0044] Using the synthetic distortion samples in the training set, the multi-attribute quality assessment model is pre-trained to obtain a pre-trained multi-attribute quality assessment model;

[0045] The pre-trained multi-attribute quality assessment model is trained on feature alignment using real distorted samples and synthetic distorted samples in the training set to obtain a trained multi-attribute quality assessment model.

[0046] For example, in order to avoid failure of model convergence during model training, which may lead to model crash, the model may be pre-trained before training the model so that the model can learn certain prior knowledge about the scoring task, thereby enabling the model to converge faster during training.

[0047] In addition, since there are usually fewer labels for real distorted samples in the training set, if real distorted samples are used to pre-train the model, it is easy to cause model overfitting. Therefore, synthetic distorted samples can be used to pre-train the model.

[0048] Accordingly, when training a multi-attribute quality assessment model, the multi-attribute quality assessment model can be pre-trained using synthetic distorted samples in the training set to obtain a pre-trained multi-attribute quality assessment model.

[0049] Exemplarily, the training of the model can be ended when the epoch (round) of pre-training the model reaches a preset number.

[0050] Among them, the model is fully trained using all the synthetic distortion samples in the training set, which can be called an epoch.

[0051] For example, since the pre-trained multi-attribute quality assessment model can only perform preliminary quality assessment on synthetic images, the quality assessment effect on real images cannot be guaranteed. Therefore, for the pre-trained multi-attribute quality assessment model, the real distorted samples and synthetic distorted samples in the training set can be used to perform feature alignment training to obtain the trained multi-attribute quality assessment model.

[0052] In one example, the feature alignment training of the pre-trained multi-attribute quality assessment model using real distorted samples and synthetic distorted samples in the training set may include:

[0053] Based on the real distorted samples and synthetic distorted samples in the training set, multiple batches of training data are generated. The number of real distorted samples in any batch of training data is consistent with the number of synthetic distorted samples.

[0054] The training data of each batch of training data is used in sequence to perform feature alignment training on the pre-trained multi-attribute quality assessment model.

[0055] For example, considering that the number of synthetic distortion samples in the training set is significantly greater than the number of real distortion samples, if the model is trained using the complete training set, it is easy to lead to poor model processing performance on real images. Therefore, in order to ensure the model's processing performance on real images, when training the pre-trained multi-attribute quality assessment model based on the real distortion samples and synthetic distortion samples in the training set, multiple batches of training data can be generated based on the real distortion samples and synthetic distortion samples in the training set.

[0056] Among them, the number of real distorted samples in the training data of any batch is consistent with the number of synthetic distorted samples.

[0057] It should be noted that the number of real distorted samples and the number of synthetic distorted samples being consistent may include that the number of real distorted samples is the same as the number of synthetic distorted samples, or the difference in number does not exceed a preset threshold.

[0058] For example, feature alignment training may be performed on the pre-trained multi-attribute quality assessment model based on training data of each batch of training data using multiple batches.

[0059] In one example, the above-mentioned feature alignment training of the pre-trained multi-attribute quality assessment model using the training data of each batch of training data in sequence may include:

[0060] For any batch of training data, perform feature alignment training on the multi-attribute quality assessment model as follows:

[0061] Phase 1: While fixing the parameters of the feature extractor in the current multi-attribute quality assessment model, the current multi-attribute quality assessment model is trained using the synthetic distortion samples in the training data of the batch;

[0062] Phase 2: While keeping the parameters of the regressor in the multi-attribute quality assessment model obtained in Phase 1 fixed, the current multi-attribute quality assessment model is trained using the real distorted samples and synthetic distorted samples in the training data of the batch;

[0063] Among them, in the initial state, the current multi-attribute quality assessment model is the pre-trained multi-attribute quality assessment model, and in the non-initial state, the current multi-attribute quality assessment model is the multi-attribute quality assessment model obtained in the second stage of the previous round.

[0064] For example, in order to enable the regressor in the multi-attribute quality assessment model to acquire the ability to quantify the quality of the synthetic image and to avoid the training process causing the feature extractor parameters to change, thereby affecting the effect of feature alignment, for any batch of training data, the synthetic distortion samples in the training data of the batch can be used to train the current multi-attribute quality assessment model while fixing the parameters of the feature extractor in the multi-attribute quality assessment model.

[0065] For example, during the training process, losses are calculated for different distortion types respectively, and then the final loss can be obtained by weighted summation.

[0066] It should be noted that different distortion types have different training difficulties, so the weights of the losses of different distortion types need to be adjusted according to the experimental results.

[0067] For example, for any batch of training data, after completing the training of stage one, in order to allow the features of the synthetic image and the real image to be aligned according to the distortion attributes and distortion degrees (even if the feature extractor treats the synthetic image and the real image equally), and to avoid the training of this process causing the parameters of the regressor to change, the parameters of the regressor in the multi-attribute quality assessment model obtained in stage one can be fixed. The real distortion samples and synthetic distortion samples in the training data of this batch are used to train the current multi-attribute quality assessment model, and the feature alignment loss is used to force the synthetic image extracted by the feature extractor to be aligned with the real image features according to the distortion degrees.

[0068] For example, during the training process, losses are calculated for different distortion types respectively, and then the final loss can be obtained by weighted summation.

[0069] It should be noted that different distortion types have different training difficulties, so the weights of the losses of different distortion types need to be adjusted according to the experimental results.

[0070] For example, in a specific implementation, stage one and stage two are executed alternately with the input of each batch of training data, that is, each time a batch of training data is input, stage one and stage two are executed in sequence.

[0071] As an example, during the training process of the above stage 2, feature alignment loss can be used for feedback optimization;

[0072] The feature alignment loss is used to characterize the difference between the first absolute difference and the second absolute difference. The first absolute difference is the absolute value of the difference between the output of the synthetic distorted sample after being processed by the multi-attribute quality assessment model and the output of the real distorted sample after being processed by the multi-attribute quality assessment model. The second absolute difference is the absolute value of the difference between the label score of the synthetic distorted sample and the label score of the real distorted sample. The smaller the difference between the first absolute value and the second absolute value, the smaller the feature alignment loss.

[0073] Exemplarily, during the training process of the above-mentioned stage 2, in order to achieve feature alignment of the real distorted samples and the synthetic distorted samples as much as possible, the feature alignment loss can be determined based on the absolute value of the difference between the output of the synthetic distorted sample after being processed by the multi-attribute quality assessment model and the output of the real distorted sample after being processed by the multi-attribute quality assessment model (referred to as the first absolute value in this article), and the absolute value of the difference between the label score of the synthetic distorted sample and the label score of the real distorted sample (referred to as the second absolute value in this article), and the feature alignment loss is used for feedback optimization.

[0074] It should be noted that the difference between the label scores of the synthetic distortion samples and the label scores of the real distortion samples refers to the difference between the label scores corresponding to the same distortion type.

[0075] For example, the feature alignment loss may be determined according to a difference between a first absolute value and a second absolute value.

[0076] Illustratively, the smaller the difference between the first absolute value and the second absolute value, the smaller the feature alignment loss.

[0077] It should be noted that, in the embodiment of the present application, the purpose of feature alignment training is to align the features of the synthetic distorted image with the features of the real distorted image, and narrow the domain difference between the two. Its specific implementation method is not limited to the method described in the above embodiment. Other methods that can achieve the above purpose (i.e., aligning the features of the synthetic distorted image with the features of the real distorted image, and narrowing the domain difference between the two), such as domain adaptation, domain alignment, etc., can also be applied to the image quality assessment scheme provided in the embodiment of the present application.

[0078] In order to enable those skilled in the art to better understand the technical solutions provided by the embodiments of the present application, the technical solutions provided by the embodiments of the present application are described below with reference to specific examples.

[0079] In this embodiment, a feature alignment training strategy is utilized to train a deep neural network (i.e., the above-mentioned multi-attribute quality assessment model) using a small number of real distorted images (i.e., the above-mentioned real distorted samples, which require manual labeling) and a large number of synthetic distorted images (i.e., the above-mentioned synthetic distorted samples, which do not require manual labeling). The deep neural network can perform quality assessment on an image based on the type of distortion contained in the image, and obtain assessment scores for a variety of different distortion types of the image.

[0080] See Figure 2 , is a schematic diagram of the overall process of image quality assessment provided by an embodiment of the present application, such as Figure 2 As shown, the overall process of image quality assessment can include: data preparation process (corresponding to the data preparation module), feature alignment training process (corresponding to the feature alignment training module) and score prediction process (corresponding to the score prediction module).

[0081] 2.1. Data preparation process, including: processing unlabeled images to obtain real distorted images and synthetic distorted images for subsequent training;

[0082] 2.2. Feature alignment training process, including: transferring knowledge learned on synthetic distorted images to real distorted images, so that only a portion of real distorted images need to be used for training, and the trained model can still achieve good results on real distorted images;

[0083] Exemplarily, the feature alignment training process may include two major processes: pre-training and feature alignment training; wherein:

[0084] 2.2.1. Pre-training process: Input synthetic images and train the model's feature extractor and regressor to enable the model to perform preliminary quality assessment of synthetic images;

[0085] 2.2.2 Feature alignment training process: See the relevant instructions below.

[0086] 2.3. Score prediction process, including: after completing model training, input the image to be tested into the model to obtain the quality scores of different distortion types (i.e., different attributes).

[0087] In this embodiment, the structure of the multi-attribute quality assessment model can be as follows: Figure 3 As shown, Figure 3 As shown, the multi-attribute quality assessment model may include: a feature extractor and a regressor.

[0088] Feature extractors typically use deep convolutional neural networks (such as ResNet (residual network)) to extract image features.

[0089] The regressor is usually composed of a fully connected layer, and after inputting the image feature vector, it will obtain the quality scores of different distortion types (attributes).

[0090] The following describes some implementation details of the above process.

[0091] 2.1 Data Preparation Process

[0092] 2.1.1. Generating synthetic images with labeled scores (i.e., obtaining synthetic distorted images): First, we can manually select (or use existing scoring models to assist) a number of high-quality images whose scores on various attributes exceed a preset score threshold. Then, we can add interference to these high-quality images, such as adjusting brightness, compressing the image, or adding noise. Finally, the degree of added interference is mapped to a label score corresponding to the distortion type.

[0093] 2.1.2. Manually label some real images (i.e., obtain real distorted images): Randomly select some real images and label them with scores based on the different distortion types.

[0094] For example, when the amount of labeled data is too small, it is necessary to manually maintain data balance to avoid a situation where images of a certain distortion type or a certain distortion degree account for too large a proportion.

[0095] 2.2 Feature Alignment Training Process

[0096] For example, Figure 4 As shown, the feature alignment training process may include: a pre-training process and a feature alignment training process.

[0097] For the pre-training process, after inputting the synthetic image, the model's feature extractor and regressor are trained, so that the model can preliminarily evaluate the quality of the synthetic image.

[0098] The feature alignment training process can be divided into two stages. The functions and implementation details of different stages include:

[0099] Phase 1 (Learning the mapping relationship from features to scores): The purpose of this phase is to train the regressor on synthetic images to gain the ability to quantify the quality of synthetic images.

[0100] Exemplarily, this stage uses synthetic distorted images with labeled scores for training, and the parameters of the feature extractor can be fixed during the training process to avoid the training process causing the feature extractor parameters to change, thereby affecting the feature alignment effect.

[0101] In addition, during the training process at this stage, different distortion types can calculate losses separately, and then the final loss can be obtained by weighted summation.

[0102] For example, since different distortion types have different training difficulties, the weights of different distortion types can be adjusted according to experimental results.

[0103] Phase 2 (fixing the regressor parameters obtained in Phase 1, extracting the common features between the real image and the synthesized image, and aligning the real image with the synthesized image): The purpose of this phase is to align the features of the synthesized image and the real image based on the distortion properties and degree of distortion (making the feature extractor treat the synthesized image and the real image equally).

[0104] In this stage, both synthetic distorted images and real distorted images are input at the same time. The batch sizes of the two are the same and they are merged into one batch and input into the model.

[0105] During the training process of this stage, the regressor parameters in the model trained in stage 1 can be fixed, and the synthetic image scores and real image scores can be output. The feature alignment loss is used to force the synthetic image extracted by the feature extractor to be aligned with the real image features according to the degree of distortion.

[0106] For example, the feature alignment loss function can be defined as follows:

[0107] Loss=|Pred s -Pred t , Label s -Label t |2

[0108] Among them, Pred s 、Pred t 、Label s and Label t are the model output of the synthetic distorted image, the model output of the real distorted image, the label score of the synthetic distorted image, and the label score of the real distorted image, respectively, and |·|2 is the mean square error.

[0109] For example, similar to stage 1, different distortion types will calculate feature alignment losses separately, and then perform weighted summation. The weights of different distortion types can also be adjusted according to experimental results.

[0110] For example, in a specific implementation, stage one and stage two are executed alternately with the input of each batch of training data, that is, each time a batch of training data is input, stage one and stage two are executed in sequence.

[0111] 2.3 Score Prediction Process

[0112] After this training, the model can evaluate the quality of both synthetically distorted images and real distorted images. Therefore, by inputting the image to be tested, the quality scores of different distortion types can be obtained.

[0113] The following describes the implementation details of the feature extractor and regressor in the multi-attribute quality assessment model.

[0114] Feature extractor: A residual network (such as ResNet18) can be used as a feature extraction network, and the hierarchical features output by its four submodules Layer1, Layer2, Layer3, and Layer4 are used as inputs for subsequent modules.

[0115] For example, in general classification and detection tasks, only the features output by the last layer are usually used, because high-level features have stronger semantic information and less noise. Unlike general classification and detection tasks, multi-attribute quality assessment tasks pay more attention to the local details of the image, and the requirements for semantics are not very high. Although the mid- and low-level features (Layer1 / Layer2 / Layer3) are less semantic and have more noise, they have higher resolution and contain more position and detail information. Therefore, the embodiment of the present application can extract all the features of each layer, fuse them (such as adding, connecting, etc.), and output them to the subsequent modules for further processing.

[0116] It should be noted that the feature extractor is not limited to using the residual network as the feature extraction network. It can also use deep convolutional networks such as VGG and DenseNet for image feature extraction.

[0117] Regressor: The regressor can mainly include N (N = number of distortion types) fully connected layers, each fully connected layer corresponds to one distortion type.

[0118] For example, the input processed feature vector will pass through each fully connected layer and output N quality scores (which can be recorded as S i ).

[0119] Wherein, i=0, 1, ..., N-1. iThe lower the value, the more severe the distortion.

[0120] It should be noted that the main purpose of the regressor is to project high-dimensional feature vectors into a score space to obtain quality scores. A fully connected layer is one possible implementation, and convolutional layers and global average pooling can also be used in specific implementations.

[0121] It can be seen that in the above process, the feature alignment training strategy is used to transfer the knowledge learned on synthetic distorted images to real distorted images. In this way, when there is less manually labeled data, a multi-attribute quality assessment model with good performance on both synthetic distorted images and real distorted images can be trained.

[0122] In addition, the model training and testing processes are both end-to-end networks. After inputting an image, it can output the scores of various distortion types, which is simple and efficient.

[0123] The multi-attribute quality assessment model obtained based on the above method can process various types of mixed distorted images, detect their distortion types, and perform quantitative evaluation on each distortion type separately.

[0124] The above describes the method provided by this application. The following describes the device provided by this application:

[0125] See Figure 5 , is a structural diagram of an image quality assessment device provided in an embodiment of the present application, such as Figure 5 As shown, the image quality assessment device may include:

[0126] A feature extraction unit 510 is configured to extract features from an input image using a feature extractor in a trained multi-attribute quality assessment model to obtain image features;

[0127] A quality assessment unit 520 is configured to use the regressor in the trained multi-attribute quality assessment model to perform score prediction on the image features to obtain assessment scores for multiple different distortion types of the input image;

[0128] The multi-attribute quality assessment model is trained using a feature alignment training method with a training set including real distorted samples and synthetic distorted samples.

[0129] In some embodiments, as Figure 6 As shown, the device also includes:

[0130] The training unit 530 is configured to train the multi-attribute quality assessment model by:

[0131] Pre-training the multi-attribute quality assessment model using the synthetic distortion samples in the training set to obtain a pre-trained multi-attribute quality assessment model;

[0132] The pre-trained multi-attribute quality assessment model is trained with feature alignment using the real distorted samples and the synthetic distorted samples in the training set to obtain a trained multi-attribute quality assessment model.

[0133] In some embodiments, the training unit 530 performs feature alignment training on the pre-trained multi-attribute quality assessment model using the real distorted samples and the synthetic distorted samples in the training set, including:

[0134] Generating multiple batches of training data based on the real distorted samples and the synthetic distorted samples in the training set, wherein the number of real distorted samples in any batch of training data is consistent with the number of synthetic distorted samples;

[0135] The pre-trained multi-attribute quality assessment model is trained on feature alignment using the training data of each batch of the multiple batches of training data in sequence.

[0136] In some embodiments, the training unit 530 sequentially uses the training data of each batch of training data to perform feature alignment training on the pre-trained multi-attribute quality assessment model, including:

[0137] For any batch of training data, perform feature alignment training on the multi-attribute quality assessment model as follows:

[0138] Phase 1: While fixing the parameters of the feature extractor in the current multi-attribute quality assessment model, the current multi-attribute quality assessment model is trained using the synthetic distortion samples in the training data of the batch;

[0139] Stage 2: While fixing the parameters of the regressor in the multi-attribute quality assessment model obtained in Stage 1, the current multi-attribute quality assessment model is trained using the real distorted samples and the synthetic distorted samples in the training data of the batch;

[0140] Among them, in the initial state, the current multi-attribute quality assessment model is the pre-trained multi-attribute quality assessment model, and in the non-initial state, the current multi-attribute quality assessment model is the multi-attribute quality assessment model obtained in stage 2 of the previous round.

[0141] In some embodiments, during the training process of the second stage, feature alignment loss is used for feedback optimization;

[0142] The feature alignment loss is used to characterize the difference between the first absolute difference and the second absolute difference, where the first absolute difference is the absolute value of the difference between the output of the synthetic distorted sample after being processed by the multi-attribute quality assessment model and the output of the real distorted sample after being processed by the multi-attribute quality assessment model, and the second absolute difference is the absolute value of the difference between the label score of the synthetic distorted sample and the label score of the real distorted sample. The smaller the difference between the first absolute value and the second absolute value, the smaller the feature alignment loss.

[0143] In some embodiments, the synthetic distortion sample is generated by adding interference to a high-quality image, the label score of each distortion type of the synthetic distortion sample is determined by the intensity of the enhanced interference, and the label score of each distortion type is negatively correlated with the intensity of the interference added to the corresponding distortion type; the high-quality image is an image whose evaluation score of each distortion type exceeds a preset threshold.

[0144] An embodiment of the present application provides an electronic device, including a processor and a memory, wherein the memory stores machine-executable instructions that can be executed by the processor, and the processor is used to execute the machine-executable instructions to implement the image quality assessment method described above.

[0145] See Figure 7 , is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. The electronic device may include a processor 701 and a memory 702 storing machine-executable instructions. The processor 701 and the memory 702 may communicate via a system bus 703. Furthermore, by reading and executing the machine-executable instructions corresponding to the image quality assessment logic in the memory 702, the processor 701 may perform the image quality assessment method described above.

[0146] The memory 702 mentioned herein may be any electronic, magnetic, optical, or other physical storage device that may contain or store information, such as executable instructions, data, and the like. For example, the machine-readable storage medium may be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, a storage drive (such as a hard disk drive), a solid-state drive, any type of storage disk (such as a CD, DVD, etc.), or similar storage media, or a combination thereof.

[0147] In some embodiments, a machine-readable storage medium is also provided. Figure 7 The memory 702 in the machine-readable storage medium stores machine-executable instructions. When executed by the processor, the machine-executable instructions implement the image quality assessment method described above. For example, the storage medium may be a ROM, RAM, CD-ROM, magnetic tape, floppy disk, or optical data storage device.

[0148] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0149] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for image quality assessment, characterized in that: include: The feature extractor in the trained multi-attribute quality assessment model is used to extract features from the input image to obtain image features; Using the regressor in the trained multi-attribute quality assessment model to perform score prediction on the image features, thereby obtaining assessment scores for multiple different distortion types of the input image; The multi-attribute quality assessment model is trained using a training set including real distorted samples and synthetic distorted samples by using feature alignment training; The multi-attribute quality assessment model is trained in the following way: Based on the real distorted samples and synthetic distorted samples in the training set, multiple batches of training data are generated. For any batch of training data, feature alignment training is performed on the pre-trained multi-attribute quality assessment model in the following manner: Phase 1: While fixing the parameters of the feature extractor in the current multi-attribute quality assessment model, the current multi-attribute quality assessment model is trained using the synthetic distortion samples in the training data of the batch; In the second stage, while fixing the parameters of the regressor in the multi-attribute quality assessment model obtained in the first stage, the current multi-attribute quality assessment model is trained using the real distorted samples and synthetic distorted samples in the training data of the batch; during the training process of the second stage, the feature alignment loss is used for feedback optimization; the feature alignment loss is used to characterize the difference between the first absolute difference and the second absolute difference, the first absolute difference is the absolute value of the difference between the output of the synthetic distorted sample after processing by the multi-attribute quality assessment model and the output of the real distorted sample after processing by the multi-attribute quality assessment model, and the second absolute difference is the absolute value of the difference between the label score of the synthetic distorted sample and the label score of the real distorted sample.

2. The method according to claim 1, characterized in that The pre-trained multi-attribute quality assessment model is trained in the following way: The multi-attribute quality assessment model is pre-trained using the synthetic distortion samples in the training set to obtain a pre-trained multi-attribute quality assessment model.

3. The method according to claim 1, characterized in that The number of real distorted samples in any batch of training data is consistent with the number of synthetic distorted samples.

4. The method according to claim 1, wherein In the initial state, the current multi-attribute quality assessment model is the pre-trained multi-attribute quality assessment model; in the non-initial state, the current multi-attribute quality assessment model is the multi-attribute quality assessment model obtained in stage 2 of the previous round.

5. The method according to claim 1, wherein The smaller the difference between the first absolute difference and the second absolute difference, the smaller the feature alignment loss.

6. The method according to any one of claims 1 to 5, characterized in that The synthetic distortion samples are generated by adding interference to high-quality images. The label scores of each distortion type of the synthetic distortion samples are determined by the strength of the enhanced interference, and the label scores of each distortion type are negatively correlated with the strength of the interference added to the corresponding distortion type. The high-quality image is an image whose evaluation scores of each distortion type exceed a preset threshold.

7. An image quality assessment device, characterized in that: include: A feature extraction unit is used to extract features from the input image using the feature extractor in the trained multi-attribute quality assessment model to obtain image features; a quality assessment unit, configured to perform score prediction on the image features using the regressor in the trained multi-attribute quality assessment model to obtain assessment scores for a plurality of different distortion types of the input image; The multi-attribute quality assessment model is trained using a training set including real distorted samples and synthetic distorted samples by using feature alignment training; The apparatus further includes: a training unit, configured to train the multi-attribute quality assessment model in the following manner: Based on the real distorted samples and synthetic distorted samples in the training set, multiple batches of training data are generated. For any batch of training data, feature alignment training is performed on the pre-trained multi-attribute quality assessment model in the following manner: Phase 1: While fixing the parameters of the feature extractor in the current multi-attribute quality assessment model, the current multi-attribute quality assessment model is trained using the synthetic distortion samples in the training data of the batch; In the second stage, while fixing the parameters of the regressor in the multi-attribute quality assessment model obtained in the first stage, the current multi-attribute quality assessment model is trained using the real distorted samples and synthetic distorted samples in the training data of the batch; during the training process of the second stage, the feature alignment loss is used for feedback optimization; the feature alignment loss is used to characterize the difference between the first absolute difference and the second absolute difference, the first absolute difference is the absolute value of the difference between the output of the synthetic distorted sample after processing by the multi-attribute quality assessment model and the output of the real distorted sample after processing by the multi-attribute quality assessment model, and the second absolute difference is the absolute value of the difference between the label score of the synthetic distorted sample and the label score of the real distorted sample.

8. The device according to claim 7, characterized in that The training unit is further configured to train the pre-trained multi-attribute quality assessment model in the following manner: Pre-training the multi-attribute quality assessment model using the synthetic distortion samples in the training set to obtain a pre-trained multi-attribute quality assessment model; The number of real distorted samples in any batch of training data is consistent with the number of synthetic distorted samples; Wherein, in the initial state, the current multi-attribute quality assessment model is the pre-trained multi-attribute quality assessment model; in the non-initial state, the current multi-attribute quality assessment model is the multi-attribute quality assessment model obtained in the second stage of the previous round; The smaller the difference between the first absolute difference and the second absolute difference, the smaller the feature alignment loss; and / or, The synthetic distortion samples are generated by adding interference to high-quality images. The label scores of each distortion type of the synthetic distortion samples are determined by the strength of the enhanced interference, and the label scores of each distortion type are negatively correlated with the strength of the interference added to the corresponding distortion type. The high-quality image is an image whose evaluation scores of each distortion type exceed a preset threshold.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores machine-executable instructions that can be executed by the processor, and the processor is configured to execute the machine-executable instructions to implement the method according to any one of claims 1 to 6.

10. A machine-readable storage medium, characterized in that The machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images