Image quality evaluation model training method based on multi-modal prompt

Abstract

The invention discloses an image quality evaluation model training method based on multi-modal prompt. The method comprises the following steps: step 1, preparing a data set required for training an image quality evaluation model; step 2, training the reference sample and the encoder of the image to be evaluated; 3, building an image quality evaluation model, taking the reference sample features and the to-be-evaluated image features as input, and taking the image quality score as output; and step 4, training an image quality evaluation model by using the data set, and performing quality evaluation on an actual to-be-evaluated image by using the trained image quality evaluation model. According to the method, a novel reference sample form is designed for a non-reference image quality evaluation model, the non-reference image quality evaluation model is composed of distorted images with different qualities and corresponding content description and appearance description, the image quality evaluation criterion is better understood, and therefore the accuracy of unknown image quality evaluation is improved.

Description

Technical Field

[0001] The present invention belongs to the field of image processing, and in particular relates to an image quality assessment model training method based on multimodal prompts. Background Art

[0002] Image quality assessment (IQA) can be divided into three types according to the availability of reference images: full-reference image quality assessment (FR-IQA), half-reference image quality assessment (RR-IQA), and no-reference image quality assessment (NR-IQA). In the first two methods, the reference image acts as a ruler and defines the standard for optimal image quality. FR-IQA and RR-IQA require that the reference image must be a high-quality image that is pixel-aligned with the image to be evaluated. By comparing the differences between the reference image and the image to be evaluated, it is easier to perform reliable and accurate image quality assessment. However, in many practical applications, it is often difficult to obtain reference images, so NR-IQA becomes a more practical choice. However, due to the limited reference information, NR-IQA often faces large uncertainties, which results in the accuracy of NR-IQA methods being generally lower than that of FR-IQA and RR-IQA methods.

[0003] To solve this problem, some works have tried to use other reference information to improve the performance of NR-IQA. For example, Liang et al. relaxed the restrictions on pixel alignment of reference images and used high-quality images with similar content as references. However, images with similar content are not always easy to obtain; Guo et al. introduced pseudo reference images, but they need to be manually selected; Yin et al. proposed a teacher-student network and a knowledge distillation method to transfer the prior knowledge in the teacher network to the student network, so that the student can use high-quality images with variable content as references. However, they still need pixel-aligned reference images to train the teacher network. Some methods use restored images as references, but the quality of restored images is not always satisfactory, which will affect the accuracy of image quality assessment. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides an image quality assessment model training method based on multimodal prompts. Different from the reference images required by FR-IQA and RR-IQA, the present invention designs a new reference sample form, which consists of distorted images of different qualities and their corresponding content descriptions and appearance descriptions. The purpose is to allow the model to have a better understanding of the image quality assessment criteria by learning the prompt information of the reference samples, thereby promoting the accuracy of the quality assessment of unknown images. The image of the reference sample is not required to be pixel-aligned with the image to be evaluated, nor does it have to have similar semantic content. At the same time, in order to allow the model to better understand the reference sample, a prompt-guided image quality assessment network is proposed, which allows the information of the reference sample to be more effectively embedded in the model, guiding the model to make a more accurate quality evaluation of the image to be evaluated.

[0005] In order to achieve the above object, the present invention adopts the following technical solution: A method for training an image quality assessment model based on multimodal prompts, the method comprising the following steps: Step 1: Prepare the data set required to train the image quality assessment model; Step 2: training encoders for reference samples and images to be evaluated, wherein the encoders are used to obtain features of the reference samples and features of the images to be evaluated, including an image encoder, a text encoder, and an appearance description encoder; Step 3: Building an image quality assessment model, taking reference sample features and image features to be assessed as input and image quality scores as output, wherein the image quality assessment model comprises a first branch consisting of a cascade of several Transformer modules and a second branch consisting of a cascade of prompt guidance modules having the same number as the Transformer modules; Step 4: Use the data set to train an image quality assessment model, and use the trained image quality assessment model to perform quality assessment on the actual image to be assessed.

[0006] Furthermore, the step 1 includes: the data set D required for training the image quality assessment model includes a number of distorted images, and each distorted image includes a quality score annotation, a content description, and an appearance description.

[0007] Furthermore, the step 2 comprises: Step 2.1, prepare M distorted images of different qualities and corresponding distorted image appearance descriptions as training sets; Step 2.2: Use CLIP pre-trained image encoder The weights are fixed and the text encoder pre-trained using CLIP is used The weights of the appearance description encoder Initialize; Step 2.3: Randomly select a distorted image in the training set and the corresponding distorted image appearance description , and -H non-corresponding distorted image appearance descriptions , the corresponding distorted image appearance description Recorded as positive sample, the appearance description of the distorted image that does not correspond Recorded as negative sample, the distorted image And positive and negative samples are input into the image encoder respectively and appearance description encoder In the output image features , positive sample features And negative sample features ; Step 2.4, calculate the loss function L: , Among them, sim represents the cosine similarity function, and exp represents the exponential function; Appearance Description Encoder The weights are updated with gradients, and the image encoder The weight of is fixed; Step 2.5, return to step 2.3 and continue training until the loss function L converges. After the training is completed, the image encoder , Text Encoder , appearance description encoder The weights are fixed.

[0008] Furthermore, the network structure of the image quality assessment model in step 3 includes: The features of the image to be evaluated are used as the input of the first branch of the image quality assessment model, and the features of the image to be evaluated and the features of the reference samples are used as the input of the second branch of the image quality assessment model. The output features of the first Transformer module of the first branch are added to the output features of the first prompt-guided module of the second branch, and then input into the next Transformer module of the first branch. At the same time, the output features of the first prompt-guided module of the second branch and the features of the reference samples are input into the next prompt-guided module of the second branch, until the output features of the last prompt-guided module are added to the output features of the last Transformer module, and then input into the regressor; wherein each prompt-guided module in the second branch includes a reference image attention unit, a content description attention unit, and an appearance description attention unit, and each unit performs a calculation of the cross-attention mechanism once; The regressor adjusts the feature addition result into a one-dimensional vector, reduces the feature dimension using a fully connected layer with a decreasing number of neurons, and outputs an image quality score; The image features to be evaluated are obtained by encoding the image to be evaluated using an image encoder, and the reference sample features are obtained by encoding distorted images of different qualities and corresponding content descriptions and appearance descriptions using an image encoder, a text encoder, and an appearance description encoder.

[0009] Further, the step 4 comprises: Step 4.1, sort all training images in data set D according to the size of the annotated quality score and divide them into C segments; Step 4.2: Randomly extract a training image and its corresponding content description and appearance description from each segment, and obtain a total of C training images and their corresponding content description and appearance description, and input them into the image encoder respectively. , Text Encoder , appearance description encoder In the reference sample feature < , , > Step 4.3: Randomly select B images from the remaining image set after extracting C training images from the dataset D as the images to be evaluated. The annotation quality score corresponding to the rth image to be evaluated is , input the image to be evaluated into the encoder In the image features to be evaluated, ; Step 4.4: The image features to be evaluated and reference sample features< , , > Input the built image quality assessment model and output the quality score of each image to be evaluated ; Step 4.5, calculate the loss function: , train the weights of the image quality assessment model, image encoder , Text Encoder , appearance description encoder The weights of remain fixed; Step 4.6: Return to step 4.2 and continue training until the loss function converges.

[0010] The beneficial effects of the present invention are: The present invention proposes a reference sample form for a no-reference image quality assessment model: using distorted images of different qualities and corresponding content descriptions and appearance descriptions as prompt information to promote the model to better understand the assessment criteria; a prompt-guided image quality assessment network is proposed, which includes several prompt-guided modules, which can effectively capture useful information in the reference samples and embed it into the model, guiding the model to make more accurate quality score predictions for the image to be evaluated based on the prompt information. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 A schematic diagram of an image quality assessment model based on multimodal prompts according to the present invention; Figure 2 This is a schematic diagram of the principle of the prompt guidance module of the present invention. DETAILED DESCRIPTION

[0012] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0013] The following are explanations of terms used in the present invention.

[0014] IQA: image quality assessment, image quality assessment; FR-IQA: full-reference IQA, full reference image quality assessment; RR-IQA: reduced-reference IQA, semi-reference image quality assessment; NR-IQA: no-reference IQA, no-reference image quality assessment; IQA dataset: Each sample in the dataset contains a distorted image and a human-annotated image quality score label. The image quality score labels are mainly in the form of MOS and DMOS. MOS: mean opinion score. When humans rate, they give a quality score to each distorted image. The higher the score, the more satisfied they are with the image. Finally, the average of the scores given by all raters is taken as the MOS of the image. DMOS: differential mean opinion score. When humans score, they compare each distorted image with its reference image and give a difference value between the two. The higher the difference value, the greater the difference in quality between the distorted image and the reference image. Finally, the average of the difference values ​​given by all scorers is taken as the DMOS of the image. Transformer: a neural network model based on attention mechanism; Cross-Attention: An attention mechanism in neural networks that calculates attention on two different inputs to handle the semantic relationship between the two inputs; PLCC: Pearson linear correlation coefficient, used to evaluate the performance of the IQA model. The larger the PLCC, the better the performance of the IQA model. SROCC: Spearman rank-order correlation coefficient, which is used to evaluate the performance of the IQA model. The larger the SROCC, the better the performance of the IQA model.

[0015] The technical solution of the present invention is described in detail below.

[0016] Given several image-text pairs as reference examples, represented as , ,in Indicates A reference image, Respectively represent The content description and appearance description corresponding to the reference image are N, which represents the number of reference samples. The reference image here does not need to be precisely aligned with the pixels of the image to be evaluated or have similar content, nor does it need to be a high-quality, distortion-free image. It can be any content and any distortion. When performing image quality assessment, the reference sample and the image to be evaluated are input into the network model together, guiding the model to predict the image quality score of the image to be evaluated under the guidance of the reference sample. Specifically: The first step is to prepare the data set required for training the model , including: a large number of distorted images of different qualities, each image has a corresponding quality score, content description and appearance description. The quality score is generally annotated by experienced experts. The content description and quality description can be generated by a generative large language model (such as ChatGPT-4V) and then corrected by humans. For example, the content description and quality description can be obtained by following the following steps: (1) Upload a distorted image to ChatGPT-4V and ask questions about the image’s content and appearance: Questions related to image content can include: What does this image show? Please describe the scene of this image. Issues related to image appearance can include: How is the quality of this image? Is there any distortion in this image? (2) The answers to each question are sorted and corrected to obtain the content description and appearance description corresponding to the image.

[0017] Step 2: Train the encoder for the reference examples and the image to be evaluated. Here, you can use a pre-trained encoder, such as the multimodal large model CLIP. CLIP consists of an image encoder and a text encoder During pre-training, CLIP trains two encoders with the goal of aligning the semantic content of images and text. Therefore, the image encoder pre-trained by CLIP can be directly used Encode the reference image and the image to be evaluated using a pre-trained text encoder Encode the content description. However, the text encoder Without pre-training with appearance descriptions, if the text encoder is used directly Encoding the appearance description cannot accurately express the meaning expressed therein. To this end, the present invention proposes a solution, using distorted images of different qualities and corresponding appearance description texts to train an appearance description encoder specifically for encoding the appearance description. , including the following steps: Step (1) prepare M distorted images of different qualities and corresponding appearance descriptions as a training set; Step (2) Use the CLIP pre-trained image encoder The weights are fixed and the text encoder pre-trained using CLIP is used The weights of the appearance description encoder Initialize; Step (3) Randomly select a distorted image in the training set and the corresponding distorted image appearance description , and H non-corresponding appearance descriptions , the corresponding appearance description is called a positive sample, and the appearance description of the distorted image that does not correspond is called a negative sample. And positive and negative samples are input into the image encoder respectively and appearance description encoder In the output image features , positive sample features And negative sample features ; Step (4) calculates the following InfoNCE loss function L: , Among them, sim represents the cosine similarity function, exp represents the exponential function, and the appearance description encoder The weights are updated with gradients, and the image encoder The weight of is fixed; Step (5) returns to step (3) and continues training until the loss function converges.

[0018] After training, the three encoders , , The weights are fixed.

[0019] Step 3: Build an image quality assessment network. Figure 1 The figure is a schematic diagram of an image quality assessment model based on multimodal prompts in the present invention. In order to better integrate the information of reference samples into the model, the present invention proposes an image quality assessment network based on prompt guidance. The network is based on the image features to be evaluated. and reference sample features< , , > is the input. The image features to be evaluated To use the image encoder The image to be evaluated Encoded, reference sample features < , , >To use the image encoder , Text Encoder , appearance description encoder Distorted images of different qualities and their corresponding content descriptions and appearance descriptions are encoded respectively, corresponding to <reference image features, content description features, appearance description features> in sequence. is the concatenation of the reference image features of each reference example along the channel. Similarly, , The content description features and quality description features of each reference sample are spliced ​​along the channel. Figure 1As shown, first the image features to be evaluated The two branches are composed of a number of Transformer modules and a number of prompt-guided modules. Each prompt-guided module consists of three parts: a reference image attention unit, a content description attention unit, and an appearance description attention unit, such as Figure 2 As shown in the figure, each unit performs a cross-attention mechanism calculation. Each unit uses the output of the previous network layer or the previous unit as the query (denoted as Q). The reference image attention unit uses the reference image features as the key (denoted as K) and value (denoted as V). Similarly, the content description attention unit and the appearance description unit use the content description features and appearance description features as K and V respectively. Then the three units perform the following calculation process in sequence: , In the formula, A represents the output feature of each unit, Softmax represents the normalized exponential function, the superscript T represents the matrix transpose, and α is a learnable parameter used to adjust the amplitude of the feature.

[0020] The first prompt guides the module to input the image features to be evaluated and reference sample features< , , >, the output features Added to the output features of the first Transformer module, and then input into the next Transformer module, outputting features at the same time The output features of the last prompt-guided module are added to the output features of the last Transformer module, and the sum of the features is input to the regressor. The regressor first adjusts the sum of the features to a one-dimensional vector, then uses a fully connected layer with a decreasing number of neurons to gradually reduce the feature dimension, and finally outputs a numerical value, namely the image quality score.

[0021] Step 4: Training the image quality assessment network, including the following steps: (1) Sort all the training images in the dataset D prepared in the first step according to the size of the annotated quality score and divide them into C segments; (2) Randomly extract a training image and its corresponding content description and appearance description from each segment, and obtain a total of C images and their corresponding content description and appearance description, which are input into the encoder pre-trained in the second step respectively. , , In the reference sample feature < , , > (3) Randomly select B images from the remaining image set after extracting C images from the dataset D as the images to be evaluated. The annotation quality score corresponding to the rth image is . Input the image to be evaluated into the encoder In the image features to be evaluated, ; (4) The image features to be evaluated and reference sample features< , , > Input the image quality assessment network built in the third step and output the quality score of each image to be evaluated ; (5) Calculate the loss function: , train the weights of the image quality assessment network, encoder , , The weights of remain fixed; (6) Return to step (2) and continue training until the loss function converges.

[0022] Step 5: Given any image to be evaluated, use the image quality evaluation network trained in step 4 for testing, including the following steps: (1) Extract an image and its corresponding content description and appearance description from each segment of the C segments of data divided in step (1), and obtain a total of C images and their corresponding content descriptions and appearance descriptions, and input them into the encoder respectively , , In the reference sample feature < , , > (2) Input the image to be evaluated into the encoder In the example above, we get the actual image features to be evaluated. ; (3) The actual image features to be evaluated and reference sample features< , , > Input the trained image quality assessment network and output the actual image quality score to be assessed , thus completing the quality assessment of an image.

[0023] Example The proposed method is used to train on the public datasets KonIQ-10k and SPAQ. The comparison results with the existing methods on the test set are shown in Table 1. The indicators for evaluating the performance of the model are the standard indicators SROCC and PLCC of the image quality assessment task. The higher the values ​​of the two indicators, the better the performance of the model.

[0024] Table 1

[0025]

[0026] In order to prove the effectiveness of each key part of the method proposed in the present invention, some components of the model are removed and retrained, and the ablation experiment results are shown in Table 2. The results show that removing any part will cause a significant decrease in model performance, thus proving the effectiveness of each key part of the proposed method in improving model performance.

[0027] Table 2

[0028]

[0029] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for training an image quality assessment model based on multimodal cues, characterized in that: The method comprises the following steps: Step 1: Prepare the data set required to train the image quality assessment model; Step 2: training encoders for reference samples and images to be evaluated, wherein the encoders are used to obtain features of the reference samples and features of the images to be evaluated, including an image encoder, a text encoder, and an appearance description encoder; Step 3: Building an image quality assessment model, taking reference sample features and image features to be assessed as input and image quality scores as output, wherein the image quality assessment model comprises a first branch consisting of a cascade of several Transformer modules and a second branch consisting of a cascade of prompt guidance modules having the same number as the Transformer modules; Step 4: Use the data set to train an image quality assessment model, and use the trained image quality assessment model to perform quality assessment on the actual image to be assessed.

2. The method for training an image quality assessment model based on multimodal prompts according to claim 1, characterized in that: The step 1 includes: the data set D required for training the image quality assessment model includes a number of distorted images, and each distorted image includes a quality score annotation, a content description, and an appearance description.

3. The method for training an image quality assessment model based on multimodal prompts according to claim 1, characterized in that: The step 2 comprises: Step 2.1, prepare M distorted images of different qualities and corresponding distorted image appearance descriptions as training sets; Step 2.2: Use CLIP pre-trained image encoder The weights are fixed and the text encoder pre-trained using CLIP is used The weights of the appearance description encoder Initialize; Step 2.3: Randomly select a distorted image in the training set and the corresponding distorted image appearance description , and H non-corresponding distorted image appearance descriptions , the corresponding distorted image appearance description Recorded as positive sample, the appearance description of the distorted image that does not correspond Recorded as negative sample, the distorted image And positive and negative samples are input into the image encoder respectively and appearance description encoder In the output image features , positive sample features And negative sample features ; Step 2.4, calculate the loss function L: ， Among them, sim represents the cosine similarity function, and exp represents the exponential function; Appearance Description Encoder The weights are updated with gradients, and the image encoder The weight of is fixed; Step 2.5, return to step 2.3 and continue training until the loss function L converges. After the training is completed, the image encoder , Text Encoder , appearance description encoder The weights are fixed.

4. The method for training an image quality assessment model based on multimodal prompts according to claim 1, characterized in that: The network structure of the image quality assessment model in step 3 includes: The features of the image to be evaluated are used as the input of the first branch of the image quality assessment model, and the features of the image to be evaluated and the features of the reference samples are used as the input of the second branch of the image quality assessment model. The output features of the first Transformer module of the first branch are added to the output features of the first prompt-guided module of the second branch, and then input into the next Transformer module of the first branch. At the same time, the output features of the first prompt-guided module of the second branch and the features of the reference samples are input into the next prompt-guided module of the second branch, until the output features of the last prompt-guided module are added to the output features of the last Transformer module, and then input into the regressor; wherein each prompt-guided module in the second branch includes a reference image attention unit, a content description attention unit, and an appearance description attention unit, and each unit performs a calculation of the cross-attention mechanism once; The regressor adjusts the feature addition result into a one-dimensional vector, reduces the feature dimension using a fully connected layer with a decreasing number of neurons, and outputs an image quality score; The image features to be evaluated are obtained by encoding the image to be evaluated using an image encoder, and the reference sample features are obtained by encoding distorted images of different qualities and corresponding content descriptions and appearance descriptions using an image encoder, a text encoder, and an appearance description encoder.

5. The method for training an image quality assessment model based on multimodal prompts according to claim 1, characterized in that: The step 4 comprises: Step 4.1, sort all training images in data set D according to the size of the annotated quality score and divide them into C segments; Step 4.2: Randomly extract a training image and its corresponding content description and appearance description from each segment, and obtain a total of C training images and their corresponding content description and appearance description, and input them into the image encoder respectively. , Text Encoder , appearance description encoder In the reference sample feature < , , > Step 4.3: Randomly select B images from the remaining image set after extracting C training images from the dataset D as the images to be evaluated. The annotation quality score corresponding to the rth image to be evaluated is , input the image to be evaluated into the encoder In the image features to be evaluated, ; Step 4.4: The image features to be evaluated and reference sample features< , , > Input the built image quality assessment model and output the quality score of each image to be evaluated ; Step 4.5, calculate the loss function: , train the weights of the image quality assessment model, image encoder , Text Encoder , appearance description encoder The weights of remain fixed; Step 4.6: Return to step 4.2 and continue training until the loss function converges.

Images