A training method for image quality assessment model based on multimodal cues

Abstract

The present invention discloses a method for training an image quality assessment model based on multimodal cues. The method comprises: step 1, preparing a data set required for training an image quality assessment model; step 2, training an encoder for a reference sample and an image to be assessed; step 3, building an image quality assessment model, taking reference sample features and image features to be assessed as input, and image quality scores as output; step 4, using the data set to train an image quality assessment model, and using the trained image quality assessment model to perform quality assessment on the actual image to be assessed. The present invention designs a new type of reference sample format for a reference-free image quality assessment model, which consists of distorted images of different qualities and their corresponding content descriptions and appearance descriptions, thereby providing a better understanding of image quality assessment criteria and promoting the accuracy of quality assessment of unknown images.

Description

Technical Field

[0001] The present invention belongs to the field of image processing, and in particular relates to a method for training an image quality assessment model based on multimodal prompts. Background Art

[0002] Image quality assessment (IQA) can be categorized into three types based on the availability of reference images: full-reference image quality assessment (FR-IQA), partial-reference image quality assessment (RR-IQA), and no-reference image quality assessment (NR-IQA). In the first two approaches, a reference image acts as a ruler, defining the standard for optimal image quality. FR-IQA and RR-IQA require that the reference image be a high-quality image that is pixel-aligned with the image being assessed. Comparing the differences between the reference image and the image being assessed facilitates reliable and accurate image quality assessment. However, in many practical applications, obtaining reference images is often difficult, making NR-IQA a more practical option. However, due to the limited reference information, NR-IQA often faces significant uncertainty, resulting in lower accuracy than FR-IQA and RR-IQA methods.

[0003] To address this issue, some works have attempted to utilize other reference information to improve the performance of NR-IQA. For example, Liang et al. relaxed the restrictions on pixel alignment of reference images and used high-quality images with similar content as references. However, images with similar content are not always easy to obtain; Guo et al. introduced pseudo reference images, but these images require manual selection; Yin et al. proposed a teacher-student network and a knowledge distillation method to transfer prior knowledge from the teacher network to the student network, enabling the student to use high-quality images with variable content as references. However, they still require pixel-aligned reference images to train the teacher network. Some methods use restored images as references, but the quality of restored images is not always satisfactory, which will affect the accuracy of image quality assessment. Summary of the Invention

[0004] In order to solve the above technical problems, the present invention provides an image quality assessment model training method based on multimodal prompts. Different from the reference images required by FR-IQA and RR-IQA, the present invention designs a new reference sample form, which consists of distorted images of different qualities and their corresponding content descriptions and appearance descriptions. The purpose is to allow the model to have a better understanding of the image quality assessment criteria by learning the prompt information of the reference samples, thereby promoting the accuracy of the quality assessment of unknown images. The image of the reference sample is not required to be pixel-aligned with the image to be evaluated, nor does it have to have similar semantic content. At the same time, in order to allow the model to better understand the reference sample, a prompt-guided image quality assessment network is proposed, which enables the information of the reference sample to be more effectively embedded in the model, guiding the model to make a more accurate quality evaluation of the image to be evaluated.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A method for training an image quality assessment model based on multimodal cues, the method comprising the following steps:

[0007] Step 1: Prepare the dataset required to train the image quality assessment model;

[0008] Step 2: Train the encoders for the reference sample and the image to be evaluated. The encoders are used to obtain features of the reference sample and the image to be evaluated, including an image encoder, a text encoder, and an appearance description encoder.

[0009] Step 3: Build an image quality assessment model that takes reference sample features and image features to be assessed as input and outputs an image quality score. The image quality assessment model includes a first branch consisting of a cascade of several Transformer modules and a second branch consisting of a cascade of prompt guidance modules, the same number of which are Transformer modules.

[0010] Step 4: Use the dataset to train an image quality assessment model, and use the trained image quality assessment model to perform quality assessment on the actual image to be assessed.

[0011] Furthermore, the step 1 includes: the dataset D required for training the image quality assessment model includes a plurality of distorted images, and each distorted image includes a quality score annotation, a content description, and an appearance description.

[0012] Furthermore, the step 2 includes:

[0013] Step 2.1: Prepare M distorted images of different qualities and their corresponding distorted image appearance descriptions as a training set.

[0014] Step 2.2: Use CLIP pre-trained image encoder The weights are fixed and the text encoder pre-trained using CLIP is used. The weights of the appearance description encoder Initialize;

[0015] Step 2.3: Randomly select a distorted image in the training set and the corresponding distorted image appearance description , and -H non-corresponding distorted image appearance descriptions , the corresponding distorted image appearance description Recorded as positive sample, the appearance description of the non-corresponding distorted image Recorded as negative samples, the distorted image And positive and negative samples are input into the image encoder respectively and appearance description encoder In the output image features , positive sample features and negative sample features ;

[0016] Step 2.4, calculate the loss function L:

[0017] ,

[0018] Among them, sim represents the cosine similarity function, and exp represents the exponential function;

[0019] Appearance Description Encoder The weights are updated with gradients, and the image encoder The weight of is fixed;

[0020] Step 2.5, return to step 2.3 and continue training until the loss function L converges. After the training is completed, the image encoder , text encoder , appearance description encoder The weights are fixed.

[0021] Furthermore, the network structure of the image quality assessment model in step 3 includes:

[0022] The image features to be evaluated are used as the input of the first branch of the image quality assessment model, and the image features to be evaluated and the reference sample features are used as the input of the second branch of the image quality assessment model. The output features of the first Transformer module of the first branch are added to the output features of the first prompt-guided module of the second branch, and then input into the next Transformer module of the first branch. At the same time, the output features of the first prompt-guided module of the second branch and the reference sample features are input into the next prompt-guided module of the second branch, until the output features of the last prompt-guided module are added to the output features of the last Transformer module and then input into the regressor. Each prompt-guided module in the second branch includes a reference image attention unit, a content description attention unit, and an appearance description attention unit, and each unit performs a cross-attention mechanism calculation.

[0023] The regressor adjusts the feature addition result into a one-dimensional vector, reduces the feature dimension using a fully connected layer with a decreasing number of neurons, and outputs an image quality score;

[0024] The image features to be evaluated are obtained by encoding the image to be evaluated using an image encoder, and the reference sample features are obtained by encoding distorted images of different qualities and corresponding content descriptions and appearance descriptions using an image encoder, a text encoder, and an appearance description encoder.

[0025] Furthermore, the step 4 includes:

[0026] Step 4.1: Sort all training images in dataset D according to their labeled quality scores and divide them into C equal segments;

[0027] Step 4.2: Randomly extract a training image and its corresponding content description and appearance description from each segment, and obtain a total of C training images and their corresponding content description and appearance description, and input them into the image encoder respectively. , text encoder , appearance description encoder In the reference sample feature < , , >

[0028] Step 4.3: Randomly select B images from the remaining image set after extracting C training images from the dataset D as the images to be evaluated. The annotation quality score corresponding to the rth image to be evaluated is , input the image to be evaluated into the encoder In the example, we get the image features to be evaluated ;

[0029] Step 4.4: The image features to be evaluated and reference sample features< , , > Input the built image quality assessment model and output the quality score of each image to be evaluated ;

[0030] Step 4.5, calculate the loss function: , train the weights of the image quality assessment model, image encoder , text encoder , appearance description encoder The weights of remain fixed;

[0031] Step 4.6. Return to step 4.2 and continue training until the loss function converges.

[0032] The beneficial effects of the present invention are:

[0033] The present invention proposes a reference sample form for a no-reference image quality assessment model: using distorted images of different qualities and corresponding content descriptions and appearance descriptions as prompt information to promote the model to better understand the evaluation criteria; a prompt-guided image quality assessment network is proposed, which includes several prompt-guided modules. The modules can effectively capture useful information in the reference samples and embed them into the model, guiding the model to make more accurate quality score predictions for the images to be evaluated based on the prompt information. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A schematic diagram of an image quality assessment model based on multimodal prompts according to the present invention;

[0035] Figure 2 This is a schematic diagram of the guiding module principle of the present invention. DETAILED DESCRIPTION

[0036] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only intended to illustrate the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0037] The following are explanations of the terms used in the present invention.

[0038] IQA: image quality assessment, image quality assessment;

[0039] FR-IQA: full-reference IQA, full reference image quality assessment;

[0040] RR-IQA: reduced-reference IQA, semi-reference image quality assessment;

[0041] NR-IQA: no-reference IQA, no-reference image quality assessment;

[0042] IQA dataset: Each sample in the dataset contains a distorted image and a human-annotated image quality score label. The image quality score labels are mainly in the form of MOS and DMOS.

[0043] MOS: Mean opinion score. When humans rate each distorted image, they assign a quality score. A higher score indicates greater satisfaction with the image's quality. The average of all raters' scores is used as the MOS for the image.

[0044] DMOS: Differential Mean Opinion Score. When humans rate, each distorted image is compared with its reference image and a difference value is assigned. The higher the difference, the greater the difference in quality between the distorted image and the reference image. The average of the difference values ​​given by all raters is used as the DMOS for the image.

[0045] Transformer: a neural network model based on the attention mechanism;

[0046] Cross-Attention: An attention mechanism in neural networks that computes attention on two different inputs to process the semantic relationship between the two inputs;

[0047] PLCC: Pearson linear correlation coefficient, used to evaluate the performance of the IQA model. The larger the PLCC, the better the performance of the IQA model.

[0048] SROCC: Spearman rank-order correlation coefficient, Spearman rank correlation coefficient, is used to evaluate the performance of the IQA model. The larger the SROCC, the better the performance of the IQA model.

[0049] The technical solution of the present invention is described in detail below.

[0050] Given several image-text pairs as reference examples, represented as , ,in Indicates the A reference image, Respectively represent The content description and appearance description corresponding to the reference image are N, which represents the number of reference samples. The reference image here does not need to be pixel-perfectly aligned with the image to be evaluated or have similar content, nor does it need to be a high-quality, undistorted image. It can have any content and any distortion. When performing image quality assessment, the reference sample and the image to be evaluated are input into the network model together, guiding the model to predict the image quality score for the image to be evaluated under the guidance of the reference sample. Specifically:

[0051] The first step is to prepare the dataset required to train the model , including: a large number of distorted images of varying quality, each with a corresponding quality score, content description, and appearance description. The quality score is typically annotated by experienced experts. The content description and quality description can be generated by a generative large language model (such as ChatGPT-4V) and then corrected by humans. For example, the content description and quality description can be obtained by following the steps below:

[0052] (1) Upload a distorted image to ChatGPT-4V and ask questions about the image’s content and appearance:

[0053] Questions related to image content can include:

[0054] What does this image show?

[0055] Please describe the scene of this image.

[0056] Issues related to image appearance can include:

[0057] How is the quality of this image?

[0058] Is there any distortion in this image?

[0059] (2) The answers to each question are sorted and corrected to obtain the content description and appearance description corresponding to the image.

[0060] Step 2: Train the encoder for the reference sample and the image to be evaluated. Here you can use a pre-trained encoder, such as the multimodal large model CLIP. CLIP consists of an image encoder and a text encoder During pre-training, CLIP trains two encoders with the goal of aligning the semantic content of images and text. Therefore, the image encoder pre-trained by CLIP can be directly used. Encode the reference image and the image to be evaluated using a pre-trained text encoder Encode the content description. However, the text encoder Without pre-training using appearance descriptions, if you use this text encoder directly Encoding the appearance description cannot accurately express the meaning expressed therein. To this end, the present invention proposes a solution, which uses distorted images of different qualities and corresponding appearance description texts to train an appearance description encoder specifically for encoding appearance descriptions. , including the following steps:

[0061] Step (1) prepare M distorted images of different qualities and corresponding appearance descriptions as training sets;

[0062] Step (2) Use the CLIP pre-trained image encoder The weights are fixed and the text encoder pre-trained using CLIP is used. The weights of the appearance description encoder Initialize;

[0063] Step (3) Randomly select a distorted image in the training set and the corresponding distorted image appearance description , and H non-corresponding appearance descriptions , the corresponding appearance description is called a positive sample, the non-corresponding distorted image appearance description is called a negative sample, and the distorted image And positive and negative samples are input into the image encoder respectively and appearance description encoder In the output image features , positive sample features and negative sample features ;

[0064] Step (4) calculates the following InfoNCE loss function L:

[0065] ,

[0066] Among them, sim represents the cosine similarity function, exp represents the exponential function, and the appearance description encoder The weights are updated with gradients, and the image encoder The weight of is fixed;

[0067] Step (5) returns to step (3) and continues training until the loss function converges.

[0068] After training, the three encoders , , The weights are fixed.

[0069] Step 3: Build an image quality assessment network. Figure 1 The figure shows a schematic diagram of an image quality assessment model based on multimodal prompts in the present invention. In order to better integrate the information of reference samples into the model, the present invention proposes an image quality assessment network based on prompt guidance. The network is based on the image features to be evaluated. and reference sample features< , , > is the input. The image features to be evaluated To use the image encoder The image to be evaluated Encoded, reference sample features < , , >To use image encoder , text encoder , appearance description encoder Distorted images of different qualities and their corresponding content descriptions and appearance descriptions are encoded respectively, corresponding to <reference image features, content description features, appearance description features> in sequence. The reference image features of each reference sample are concatenated along the channel. Similarly, , The content description features and quality description features of each reference sample are spliced ​​along the channel. Figure 1 As shown, first the image features to be evaluated Enter into two branches. The two branches are composed of a number of Transformer modules cascade and a number of prompt guidance modules cascaded with the same number of Transformer modules. Each prompt guidance module consists of three parts: a reference image attention unit, a content description attention unit, and an appearance description attention unit, such as Figure 2 As shown in the figure, each unit performs a cross-attention calculation. Each unit uses the output of the previous network layer or unit as the query (denoted as Q). The reference image attention unit uses the reference image features as the key (denoted as K) and value (denoted as V). Similarly, the content description attention unit and appearance description unit use the content description features and appearance description features as K and V, respectively. Then, the three units perform the following calculation process in sequence:

[0070] ,

[0071] Where A represents the output feature of each unit, Softmax represents the normalized exponential function, the superscript T represents the matrix transpose, and α is a learnable parameter used to adjust the amplitude of the feature.

[0072] The first prompt guides the module to input the image features to be evaluated and reference sample features< , , >, will output features Added to the output features of the first Transformer module, and then input into the next Transformer module, outputting features at the same time This is then fed into the next cue-guided module. After the output features of the last cue-guided module are added to the output features of the last Transformer module, the sum of these features is fed into the regressor. The regressor first resizes the sum of these features into a one-dimensional vector. It then uses fully connected layers with decreasing numbers of neurons to gradually reduce the feature dimensionality. Finally, it outputs a single value, the image quality score.

[0073] Step 4: Training the image quality assessment network, including the following steps:

[0074] (1) Sort all the training images in the dataset D prepared in the first step according to the size of the labeled quality scores and divide them into C segments;

[0075] (2) Randomly extract a training image and its corresponding content description and appearance description from each segment, and obtain a total of C images and their corresponding content description and appearance description, which are input into the encoder pre-trained in the second step respectively. , , In the reference sample feature < , , >

[0076] (3) Randomly select B images from the remaining image set after extracting C images from the dataset D as the images to be evaluated. The annotation quality score corresponding to the rth image is . Input the image to be evaluated into the encoder In the example, we get the image features to be evaluated ;

[0077] (4) The image features to be evaluated and reference sample features< , , > Input the image quality assessment network built in the third step and output the quality score of each image to be evaluated ;

[0078] (5) Calculate the loss function: , train the weights of the image quality assessment network, encoder , , The weights of remain fixed;

[0079] (6) Return to step (2) and continue training until the loss function converges.

[0080] Step 5: Given any image to be evaluated, use the image quality assessment network trained in step 4 to perform a test application, including the following steps:

[0081] (1) Extract an image and its corresponding content description and appearance description from each segment of the C segment data divided in step 4 (1), and obtain a total of C images and their corresponding content description and appearance description, and input them into the encoder respectively , , In the reference sample feature < , , >

[0082] (2) Input the image to be evaluated into the encoder In the example above, we get the actual image features to be evaluated. ;

[0083] (3) The actual image features to be evaluated and reference sample features< , , >Input the trained image quality assessment network and output the actual image quality score to be assessed , thus completing the quality assessment of an image.

[0084] Example

[0085] The proposed method was trained on the public datasets KonIQ-10k and SPAQ. Table 1 shows the comparison results with existing methods on the test set. The model performance was evaluated using the standard image quality assessment metrics, SROCC and PLCC. Higher values ​​for these two metrics indicate better model performance.

[0086] Table 1

[0087]

[0088] To demonstrate the effectiveness of the key components of the proposed method, some components of the model were removed and retrained. The ablation experiment results are shown in Table 2. The results show that removing any component will cause a significant decrease in model performance, thus proving the effectiveness of the key components of the proposed method in improving model performance.

[0089] Table 2

[0090]

[0091] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for training an image quality assessment model based on multimodal cues, characterized in that: The method comprises the following steps: Step 1: Prepare the dataset required to train the image quality assessment model; Step 2: Train the encoders for the reference sample and the image to be evaluated. The encoders are used to obtain features of the reference sample and the image to be evaluated, including an image encoder, a text encoder, and an appearance description encoder. This includes: Step 2.1: Prepare M distorted images of different qualities and their corresponding distorted image appearance descriptions as a training set. Step 2.2: Use CLIP pre-trained image encoder The weights are fixed and the text encoder pre-trained using CLIP is used. The weights of the appearance description encoder Initialize; Step 2.3: Randomly select a distorted image in the training set and the corresponding distorted image appearance description , and H non-corresponding distorted image appearance descriptions , the corresponding distorted image appearance description Recorded as positive sample, the appearance description of the non-corresponding distorted image Recorded as negative samples, the distorted image And positive and negative samples are input into the image encoder respectively and appearance description encoder In the output image features , positive sample features and negative sample features ; Step 2.4, calculate the loss function L: ， Among them, sim represents the cosine similarity function, and exp represents the exponential function; Appearance Description Encoder The weights are updated with gradients, and the image encoder The weight of is fixed; Step 2.5, return to step 2.3 and continue training until the loss function L converges. After the training is completed, the image encoder , text encoder , appearance description encoder The weights of are fixed; Step 3: Build an image quality assessment model that takes reference sample features and image features to be assessed as input and outputs an image quality score. The image quality assessment model includes a first branch consisting of a cascade of several Transformer modules and a second branch consisting of a cascade of prompt guidance modules, the same number of which are Transformer modules. The network structure of the image quality assessment model includes: The image features to be evaluated are used as the input of the first branch of the image quality assessment model, and the image features to be evaluated and the reference sample features are used as the input of the second branch of the image quality assessment model. The output features of the first Transformer module of the first branch are added to the output features of the first prompt-guided module of the second branch, and then input into the next Transformer module of the first branch. At the same time, the output features of the first prompt-guided module of the second branch and the reference sample features are input into the next prompt-guided module of the second branch, until the output features of the last prompt-guided module are added to the output features of the last Transformer module and then input into the regressor. Each prompt-guided module in the second branch includes a reference image attention unit, a content description attention unit, and an appearance description attention unit, and each unit performs a cross-attention mechanism calculation. The regressor adjusts the feature addition result into a one-dimensional vector, reduces the feature dimension using a fully connected layer with a decreasing number of neurons, and outputs an image quality score; The features of the image to be evaluated are obtained by encoding the image to be evaluated using an image encoder. The features of the reference sample are obtained by encoding distorted images of different qualities and their corresponding content descriptions and appearance descriptions using an image encoder, a text encoder, and an appearance description encoder respectively. Step 4: Use the dataset to train an image quality assessment model, and use the trained image quality assessment model to perform quality assessment on the actual image to be assessed.

2. The method for training an image quality assessment model based on multimodal prompts according to claim 1, characterized in that: The step 1 includes: the dataset D required for training the image quality assessment model includes a plurality of distorted images, and each distorted image includes a quality score annotation, a content description, and an appearance description.

3. The method for training an image quality assessment model based on multimodal prompts according to claim 1, characterized in that: The step 4 comprises: Step 4.1: Sort all training images in dataset D according to their labeled quality scores and divide them into C equal segments; Step 4.2: Randomly extract a training image and its corresponding content description and appearance description from each segment, and obtain a total of C training images and their corresponding content description and appearance description, and input them into the image encoder respectively. , text encoder , appearance description encoder In the reference sample feature < , , > Step 4.3: Randomly select B images from the remaining image set after extracting C training images from the dataset D as the images to be evaluated. The annotation quality score corresponding to the rth image to be evaluated is , input the image to be evaluated into the encoder In the example, we get the image features to be evaluated ; Step 4.4: The image features to be evaluated and reference sample features< , , > Input the built image quality assessment model and output the quality score of each image to be evaluated ; Step 4.5, calculate the loss function: , train the weights of the image quality assessment model, image encoder , text encoder , appearance description encoder The weights of remain fixed; Step 4.

6. Return to step 4.2 and continue training until the loss function converges.

Images