Visual language model-based few-sample image quality evaluation method and device

Abstract

The invention discloses a few-sample image quality evaluation method and device based on a visual language model, and relates to the technical field of image quality evaluation. Gradient adjustment meta-prompt image quality evaluation (GRMP-IQA) is introduced, meta-knowledge from various image distortions is merged by adopting a double-layer gradient optimization process through meta-prompt pre-training, so that text refining and initialization of visual prompt are realized, the adaptive capacity of a CLIP model to an IQA task is enhanced, and the accuracy of the image quality evaluation is improved. The sensitivity of prompt initialization to various IQA scenes is reduced; and then correcting the deviation introduced by excessive attention of semantic content of a visual-language model (CLIP) through quality perception gradient regularization, and ensuring that the image quality is evaluated more accurately, so that the pre-trained CLIP model adapts to a BIQA task with a small number of training samples.

Description

Technical Field

[0001] The present invention relates to the technical field of image quality assessment, and in particular to a method and device for assessing the quality of a few-sample image based on a visual language model. Background Art

[0002] With the advent of the mobile Internet era, the focus on computer vision has shifted from initial compression and image processing to processing user-generated content such as smartphone photos and videos, and more recently, AI-generated content. This progress has accelerated a significant escalation in the demand for effective image quality assessment (IQA) techniques, emphasizing the importance of developing methods that can proficiently assess image quality without reference images. In the field of image quality assessment, methods are roughly divided into: full-reference IQA (FR-IQA) with complete reference images, reduced-reference IQA (RR-IQA) with partial reference data, and blind IQA (BIQA) without any reference images, depending on the reference information used. Due to the lack of reference images, BIQA poses a unique and challenging problem in practical applications.

[0003] Recent BIQA methods attempt to alleviate this problem, with DEIQT introducing a transformer decoder to help the visual transformer (ViT) adapt to the IQA task. In addition, some self-supervised methods exploit contrastive learning during pre-training to extract generic quality features, which may improve performance even with limited label availability. Data-driven BIQA models based on deep neural networks have made significant progress in recent years. However, the quality scores of distorted images are typically measured using the mean opinion score (MOS), which is the average of multiple ratings (sometimes as high as 120). However, these methods typically require large pre-training datasets and involve training large-scale neural networks with millions of parameters. Therefore, obtaining a sufficient number of IQA training samples is a very time-consuming and labor-intensive task, resulting in a large training cost.

[0004] Recently, pre-trained visual-language models (CLIP) have achieved surprising performance on various downstream tasks. In particular, CLIP-IQA shows promising zero-shot generalization capabilities by simply filling in handcrafted hints (e.g., “good photo”) as the input to the CLIP encoder in IQA tasks. Despite these advances, such models may encounter a domain gap between the upstream training data and the specific requirements of the IQA downstream task, which limits the performance of zero-shot methods. Therefore, data-efficient IQA is expected to provide a more robust and effective way to predict image quality compared to zero-shot methods, but remains relatively unexplored. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a few-sample image quality assessment method and device based on a visual language model, by introducing gradient regulated meta-prompted image quality assessment (GRMP-IQA), so that a pre-trained visual-language model (CLIP) is adapted to BIQA tasks with a small number of training samples.

[0006] In a first aspect, the present invention provides a method for evaluating image quality of a small number of samples based on a visual language model, comprising:

[0007] Meta-cue pre-training process: In the distorted dataset, the support dataset and the query dataset as the meta-knowledge task are divided according to the distortion type; in the inner loop process, the features are extracted through the partially learnable visual encoder and the frozen text encoder according to the input support dataset, and the quality score of the distorted image is output according to the visual-text feature similarity, and then the training loss is calculated according to the quality label corresponding to the image to optimize the meta-cue, which includes the quality visual cue and the quality text cue. The quality text cue is used as the pre-trained CLIP model V qua The input of the text encoder is the quality visual cues as the pre-trained CLIP model V qua In the outer loop, the query dataset is used to optimize the meta-prompt according to the inner loop process, and the pre-trained CLIP model V is obtained. qua ;

[0008] Quality-aware gradient regularization process: images, quality text hints, and semantic hints are passed to the pre-trained CLIP model V qua Image encoder and text encoder and semantic CLIP model V sem , quantify the pre-trained CLIP model V by calculating the KL divergence between zero-shot semantic prediction probabilities qua The quality optimization direction of the IQA task is obtained by calculating the cross entropy between the model’s prediction and the basic facts. Finally, the IQA task gradient is adjusted by clipping the gradient aligned with the general semantic direction in the shared representation space, and the regularized gradient is generated for backward updating to balance the learning between quality and semantics.

[0009] In a second aspect, the present invention provides a device for evaluating image quality of a small number of samples based on a visual language model, comprising:

[0010] The meta-cue pre-training module is used to divide the support dataset and the query dataset as the meta-knowledge task in the distorted dataset according to the distortion type; in the inner loop process, the features are extracted through the partially learnable visual encoder and the frozen text encoder according to the input support dataset, and the quality score of the distorted image is output according to the visual-text feature similarity, and then the training loss is calculated according to the quality label corresponding to the image to optimize the meta-cue, and the meta-cue includes the quality visual cue and the quality text cue. The quality text cue is used as the pre-trained CLIP model V qua The input of the text encoder is the quality visual cues as the pre-trained CLIP model V qua In the outer loop, the query dataset is used to optimize the meta-prompt according to the inner loop process, and the pre-trained CLIP model V is obtained. qua ;

[0011] The quality-aware gradient regularization module is used to pass images, quality text cues, and semantic cues into the pre-trained CLIP model V qua Image encoder and text encoder and semantic CLIP model V sem , quantify the pre-trained CLIP model V by calculating the KL divergence between zero-shot semantic prediction probabilities qua The quality optimization direction of the IQA task is obtained by calculating the cross entropy between the model’s prediction and the basic facts. Finally, the IQA task gradient is adjusted by clipping the gradient aligned with the general semantic direction in the shared representation space, and the regularized gradient is generated for backward updating to balance the learning between quality and semantics.

[0012] The technical solution provided by the present invention has at least the following technical effects:

[0013] By introducing Gradient Regulated Meta-Prompted Image Quality Assessment (GRMP-IQA), meta-cue pre-training is first used to merge meta-knowledge from various image distortions using a two-layer gradient optimization process, which is then used to refine the initialization of text and visual cues, thereby enhancing the adaptability of the CLIP model to IQA tasks and reducing the sensitivity of prompt initialization to various IQA scenarios. Quality-aware gradient regularization is then used to correct the bias introduced by the visual-language model (CLIP) that over-focuses on semantic content, ensuring a more accurate assessment of image quality and making the pre-trained CLIP model adaptable to BIQA tasks with a small number of training samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The present invention will be further described below in conjunction with embodiments with reference to the accompanying drawings.

[0015] Figure 1Schematic diagram of the overall process of the method in Embodiment 1 of the present invention;

[0016] Figure 2 Schematic diagram of the principle of gradient adjustment element prompt image quality assessment in the first embodiment of the present invention;

[0017] Figure 3 It is a schematic diagram comparing the gMAD evaluation results of DEIQT in Example 2 of the present invention and GRMP-IQA of the present invention;

[0018] Figure 4 It is a schematic diagram of the structure of the device in the second embodiment of the present invention. DETAILED DESCRIPTION

[0019] The embodiments of the present invention provide a few-sample image quality assessment method and apparatus based on a visual language model, and introduce a gradient regulated meta-prompted image quality assessment (GRMP-IQA) to adapt a pre-trained visual-language model (CLIP) to BIQA tasks with a small number of training samples.

[0020] The technical solution in the embodiment of the present invention has the following general idea:

[0021] Gradient Modulated Meta-Prompted Image Quality Assessment (GRMP-IQA) was introduced, such as Figure 2 As shown in Figure 1, GRMP-IQA consists of two main components: meta-prompt pre-training module (MPP) and quality-aware gradient regularization (QGR), that is, the present invention includes a meta-prompt training stage and a quality-aware gradient regularization stage. Specifically, the meta-prompt training stage includes two parts: 1. Using the support dataset D in the inner loop support Optimize; 2. Use query data set D in the outer loop query To this end, we first divide the existing large-scale distortion dataset into a support dataset D according to the distortion type as the meta-knowledge task. support And query dataset D query , and pass them into the inner and outer loops respectively. In the inner loop, the present invention uses D support The input image in is extracted by partially learnable visual encoder and frozen text encoder respectively, and the quality score of the distorted image is output according to the feature similarity of visual and text, and the training loss is calculated according to the quality label corresponding to the image to optimize the learnable meta-prompt (including quality visual prompt and quality text prompt) in the present invention. In the outer loop process, based on the meta-prompt updated in the inner loop, the query dataset D is continued to be used. queryFollowing the inner loop process, the meta-cues are trained. This process effectively simulates the process of the model generalizing different scenarios and effectively learns the quality-aware priors shared by different scenarios. In the quality-aware gradient regularization stage, the image, semantic cues, and learnable quality text cues are respectively passed to the image encoder (where the quality visual cues obtained in the first stage of training are used as part of the image encoder parameters), the text encoder, and the semantic CLIP, so as to calculate the CE loss and the KL loss, which represent the IQA task direction and the general semantic direction, respectively. In the gradient regularization process, the IQA task direction is refined by clipping the gradients aligned with the general semantic direction, thereby ensuring that the model prioritizes image quality and achieves accurate evaluation of image quality. The meta-cue training stage (MPP) focuses on pre-training of visual text cues and uses shared meta-knowledge about distortions to promote rapid adaptation to various IQA tasks. The quality-aware gradient regularization stage (QGR) plays a key role in the fine-tuning process by adjusting the update gradient to mitigate overfitting to semantic content. The specific stages are shown below.

[0022] Meta-cue training phase: randomly extract a small batch of samples from the meta-knowledge task and divide it into support set D support and query set D query A two-layer gradient descent method from D support The inner loop on advances to D query The outer loop on θ optimizes the learnable meta-cue [θ T ,θ V ], where θ T represents the quality text prompt, θ V Representing quality visual cues, these cues are then used as initial conditions in the fine-tuning phase.

[0023] Quality-aware gradient regularization stage: Given an input image x and a semantic cue w i = "a photo of a[class].", predict the class probability distribution p of image x using the original semantic CLIP model and the pre-trained CLIP model sem (w i |x) and p qua (w i |x), and calculate the gradient of the KL divergence loss As a measure of general semantic direction. At the same time, according to the quality text hint θ T and image x, calculate the quality loss gradient between the predicted quality and the actual quality label Finally, by clipping and general semantic direction G sem Aligned gradients to refine the IQA task direction G qua →G qgrThis approach ensures that the model prioritizes image quality, thereby minimizing the impact of semantic noise.

[0024] 1. Meta Tips

[0025] In this embodiment, a meta-hint is proposed, which includes two learnable hints, one is a quality visual hint and the other is a quality text hint, where the quality text hint is used as the input of the text encoder and the quality visual hint is used as part of the image encoder parameters. The meta-hint combines the adjustment of text and visual hints to work in a mutually reinforcing way. The goal is to enable these learnable hints to collect shared knowledge in different distortion scenarios, thereby promoting the rapid adaptation of CLIP to the IQA field.

[0026] 1.1. Visual cues of quality

[0027] Deep Prompt Tuning (DPT) is adopted as the visual meta-cue. Formally, the input embedding of the self-attention module of layer l of the vit-based image encoder in CLIP is represented as where f l represents the classification label embedding, Represents image patch embeddings; deep cue fine-tuning will learn the labels The above token sequence added to each ViT layer in the CLIP image encoder; the multi-head attention module of layer l then processes the input tokens as follows:

[0028]

[0029] Among them, P l The output embedding of is discarded, denoted as -, and is not fed to subsequent layers.

[0030] 1.2. Quality text prompts

[0031] The text meta-cue context is optimized by modeling the context words as continuous vectors, and end-to-end learning is achieved while the pre-training parameters are frozen. Specifically, the quality text cue is defined as q i ={v1,v2,…,v M ,[quality]}, where q i Indicates i th The learnable text features of the class, v M represents a learnable vector, embedding the quality label [quality]∈{high quality,low quality}. In the GRMP-IQA of this embodiment, the length of the vector is set to 4.

[0032] 2. Meta-cue pre-training

[0033] As mentioned before, during the pre-training phase of cue optimization, the focus is only on optimizing the learnable meta-cue, defined as θ = [θ T ,θ V ]. Among them, θ V is a visual cue of quality, θ T is the quality textual cue. In this case, for an input image x, the textual meta-cue q is processed by a text encoder g(·) i ; At the same time, the image x is encoded by the visual encoder and the [CLS] tag is used to extract the feature vector f; therefore, the probability of predicting high quality is mathematically expressed as:

[0034]

[0035] Among them, q high and q low denote high-quality and low-quality category cues, respectively, p(q high |x) represents the probability that the estimated image quality x is high quality, τ is the temperature parameter learned by CLIP, and <,> represents the cosine similarity;

[0036] Then, the labeled quality scores are rescaled to 0-1, recorded as y, and the loss function is calculated as:

[0037] L=-(ylog(p(q high |x))+(1-y)log(1-p(q high |x))) (3).

[0038] 2.1. Constructing Distortion Meta-knowledge Task

[0039] The ability to detect various types of image distortions is crucial for developing BIQA models with strong generalization capabilities across different scenarios. In addition, the effectiveness of prompt tuning depends heavily on the initial configuration of the prompts. This initial setting greatly affects the ability of the CLIP visual-language model to quickly adapt to different IQA scenarios. Inspired by the "learning to learn" spirit inherent in the deep meta-learning paradigm, this paper proposes an optimization-based method for effectively pre-training visual-text prompts. These prompts incorporate shared quality insights from various image distortions, thereby enhancing CLIP's rapid adaptability to IQA tasks. In order to study the general laws of image distortion, we first construct a K t -way task, denoted as T t , and then use it to construct the meta-training set in, and denote the support dataset and query dataset of each task, respectively, and T denotes the total number of tasks. In order to simulate the fast generalization process to different distortions in the BIQA scenario, K tasks are randomly sampled from the meta-training set 1≤k≤T as mini-batches to perform two-layer gradient optimization.

[0040] 2.2 Distortion-aware Meta-cue Learning

[0041] The method adopts a two-layer gradient descent technique to connect the learning process from the support set to the query set, thereby promoting the acquisition of quality prior knowledge through prompts. Specifically, it mainly consists of two optimization steps. In the inner loop (first layer), the gradient of the prompt parameters is calculated using the support set and the first update is applied. In the outer loop (second layer), the performance of the updated model on the query set is evaluated. This two-layer structure enables the prompts to quickly gain generalization capabilities across different BIQA contexts by optimizing the meta-prompts for each unique distortion scenario.

[0042] 2.2.1. Inner loop process

[0043] The goal is to make the meta-cue θ = [θ T ,θ V ]Adaptation support dataset During the first level update, the loss is determined The model parameters are then updated on the support set using an internal learning rate α, specified as:

[0044]

[0045] 2.2.2 External circulation process

[0046] Similarly, the second level updates the query set Adjust model parameters θ i ′:

[0047]

[0048] Aggregate the gradients from all tasks to update the final model parameters, following the update rule:

[0049]

[0050] Here, β represents the external learning rate.

[0051] This methodical selection and optimization of tasks from the meta-training set effectively trains a learnable meta-cue θ = [θ T ,θ V ], thus ensuring their generalization across various image distortions.

[0052] 3. Quality-aware gradient regularization

[0053] The original CLIP model was pre-trained on the classification task to obtain the semantic CLIP model, denoted as V sem The CLIP model pre-trained in the meta-prompt pre-training module is used by V qua express.

[0054] Recent studies have highlighted the inconsistency between low-level tasks of IQA and high-level semantic representations from various upstream tasks. In this case, the model may over-focus on semantic content at the expense of image quality, resulting in overfitting and decreased generalization ability. To alleviate this problem, this embodiment proposes a quality-aware gradient regularization (QGR) technique. This method uses the semantic CLIP model V sem Directional guidance for zero-shot semantic prediction in the pre-trained CLIP model V qua Refine the quality gradient updates during fine-tuning on the IQA task. The goal is to reduce the over-emphasis on semantics and promote a balanced assessment of image quality.

[0055] Specifically, in order to obtain the direction of general semantic knowledge G sem First, a hard prompt w is designed i = aphoto of a[class], where [class] belongs to one of nine semantic categories: {"animal", "cityscape", "person", "indoor", "landscape", "night", "plant", "still life", "other"}. i Input to CLIP model V sem and V qua The text encoder extracts text features and compares them with the model V sem and V qua Align the image features extracted by the visual encoding extractor in to obtain the zero-sample semantic prediction probability p sem (w i |x) and p qua (w i |x). For the pre-trained CLIP model V qua The goal of fine-tuning is not to over-focus on semantic information. Therefore, by calculating p sem (w i |x) and p qua (w i |x) to quantify the fine-tuning CLIP model V qua The degree of semantic attention in:

[0056]

[0057] Instead, by calculating the model's prediction p(q high The cross entropy L between |x) and the ground truth yce (V qua ) obtains the quality optimization direction in the IQA field, as described in Formula 3. The learning between quality and semantics is balanced by adjusting the gradient of the task in the shared representation space.

[0058] Specifically, the gradient Represented as G qua , Represented as G sem , the relationship between them can be described in two aspects:

[0059] (1) If G qua With G sem The angle between G and the semantic direction is less than 90 degrees, indicating that the optimization direction of small sample IQA quality knowledge is consistent with that of general semantic knowledge. sem Parallel components G || Shear G qua , to adjust the original quality-aware optimization path of the model to prevent overfitting on semantic relevance.

[0060] (2) If G qua With G sem The angle between them is greater than or equal to 90 degrees, indicating that they are in different directions during the improvement process. In this case, the mass gradient G is not adjusted. qua , allowing the model to learn quality-aware features based on the original optimization direction.

[0061] The regularization strategy is mathematically expressed as follows:

[0062]

[0063] The introduction of λ to generalize the formula provides flexibility in the degree of guidance of general knowledge in control applications. Specifically, λ = 1 means that G qua Projection to G sem in the orthogonal direction.

[0064] The contributions of the technical solution of this embodiment are summarized as follows:

[0065] (1) We propose a meta-cue pre-training module that organizes meta-tasks by image distortion type and optimizes soft cues via double-layer gradient descent. This approach allows soft cues to capture shared meta-knowledge across different distortions, greatly improving the model’s rapid adaptability to a wide range of IQA scenarios.

[0066] (2) We propose a novel quality-aware gradient regularization method to adjust the update gradients during fine-tuning to maintain a balance between semantic and quality information. This method clips the gradients that are consistent with the semantic direction, ensuring that the model prioritizes image quality while integrating semantic context where relevant.

[0067] (3) Furthermore, the integration of textual and visual cue fine-tuning is promoted in a synergistic manner, thereby enhancing their mutual effectiveness and providing superior generalization in a data-efficient setting.

[0068] Embodiment 1

[0069] This embodiment provides a few-sample image quality assessment method based on a visual language model. Figure 1 As shown, including:

[0070] Meta-cue pre-training process: In the distorted dataset, the support dataset and the query dataset as the meta-knowledge task are divided according to the distortion type; in the inner loop process, the features are extracted through the partially learnable visual encoder and the frozen text encoder according to the input support dataset, and the quality score of the distorted image is output according to the visual-text feature similarity, and then the training loss is calculated according to the quality label corresponding to the image to optimize the meta-cue, which includes the quality visual cue and the quality text cue. The quality text cue is used as the pre-trained CLIP model V qua The input of the text encoder is the quality visual cues as the pre-trained CLIP model V qua In the outer loop, the query dataset is used to optimize the meta-prompt according to the inner loop process, and the pre-trained CLIP model V is obtained. qua ;

[0071] Quality-aware gradient regularization process: images, quality text hints, and semantic hints are passed to the pre-trained CLIP model V qua Image encoder and text encoder and semantic CLIP model V sem , quantify the pre-trained CLIP model V by calculating the KL divergence between zero-shot semantic prediction probabilities qua The quality optimization direction of the IQA task is obtained by calculating the cross entropy between the model’s prediction and the basic facts. Finally, the IQA task gradient is adjusted by clipping the gradient aligned with the general semantic direction in the shared representation space, and the regularized gradient is generated for backward updating to balance the learning between quality and semantics.

[0072] In a possible implementation, during the meta-prompt pre-training process, a support dataset and a query dataset as the meta-knowledge task are divided in the distorted dataset according to the distortion type, specifically including: constructing a K t -way task, denoted as T t , and then use it to construct the meta-training set in, and denote the support dataset and query dataset of each task respectively, and T denotes the total number of tasks. K tasks are randomly sampled from the meta-training set 1≤k≤T as mini-batches to perform two-layer gradient optimization.

[0073] During the meta-prompt pre-training process, for an input image x, the textual meta-prompt q is processed by the text encoder g(·) i ; At the same time, the image x is encoded by the visual encoder and the [CLS] tag is used to extract the feature vector f; therefore, the probability of predicting high quality is mathematically expressed as:

[0074]

[0075] Among them, q high and q low denote high-quality and low-quality category cues, respectively, p(q high |x) represents the probability that the estimated image quality x is high quality, τ is the temperature parameter learned by CLIP, and <,> represents the cosine similarity;

[0076] Then, the labeled quality scores are rescaled to 0-1, recorded as y, and the loss function is calculated as:

[0077] L=-(ylog(p(q high |x))+(1-y)log(1-p(q high |x))).

[0078] The inner loop process is used to make the meta-hint θ = [θ T ,θ V ]Adaptation support dataset Determine the loss first The model parameters are then updated on the support set using an internal learning rate α, specified as:

[0079]

[0080] The outer loop process is used to Adjust model parameters θ i ′:

[0081]

[0082] Aggregate the gradients from all tasks to update the final model parameters, following the update rule:

[0083]

[0084] Here, β represents the external learning rate.

[0085] The quality visual cue adopts deep cue tuning as the visual meta-cue, and the input embedding of the self-attention module of the l-layer of the vit-based image encoder in CLIP is represented as where f l represents the classification label embedding, Represents image patch embeddings; deep cue fine-tuning will learn the labels The above token sequence added to each ViT layer in the CLIP image encoder; the multi-head attention module of layer l then processes the input tokens as follows:

[0086]

[0087] Among them, P l The output embedding of is discarded, denoted as -, and is not fed to subsequent layers;

[0088] The quality text hint is defined as q i ={v1,v2,...,v M ,[quality]}, where q i Indicates i th The learnable text features of the class, v M Represents a learnable vector, embedding the quality label [quality]∈{high quality,lowquality}.

[0089] The quality-aware gradient regularization process specifically includes: given an input image x and a semantic hint w i = "aphoto ofa[class].", predict the class probability distribution p of image x using the semantic CLIP model and the pre-trained CLIP model sem (w i |x) and p qua (w i |x), and calculate the gradient of the KL divergence loss ▽L kl As a measure of general semantic direction; the formula for calculating KL divergence loss is:

[0090]

[0091] At the same time, according to the quality text prompt θ T and image x, calculate the quality loss gradient between the predicted quality and the actual quality label As the quality direction; finally, by cutting and general semantic direction G sem Aligned gradients to refine the IQA task direction G qua →G qgr .

[0092] Cropping and general semantic direction G semAligned gradients specifically include:

[0093] The gradient Represented as G qua , Represented as G sem , if G qua With G sem If the angle between the two directions is less than 90 degrees, then the sem Parallel components G || Shear G qua , if G qua With G sem If the angle between the two is greater than or equal to 90 degrees, the mass gradient G is not adjusted. qua ; The regularization strategy is expressed as follows:

[0094]

[0095] Among them, λ=1 means that G qua Projection to G sem in the orthogonal direction.

[0096] Figure 3 Schematic diagram comparing the gMAD evaluation results of DEIQT and GRMP-IQA of the present invention. (a) DEIQT is fixed at a low quality level. (b) DEIQT is fixed at a high quality level. (c) The present invention is fixed at a low quality level. (d) The present invention is fixed at a high quality level.

[0097] Extensive experimental results highlight the effectiveness and efficiency of the method of this embodiment. It is worth noting that with only 200 data samples, the proposed method surpasses the SOTA model on the CLIVE dataset (20% of the training data) and achieves highly competitive performance on the KonIQ dataset (2% of the training data), as shown in Tables 1-4, where CLIVE and KONIQ are the dataset names, and the number of pictures used for training is after the underscore; the first column is the name of a representative image quality evaluation method, such as TReS, which uses a hybrid architecture with a convolutional neural network and a Transformer network for image quality prediction. The bold data entries represent the best results.

[0098] Table 1 Performance comparison of real database under few-sample conditions, measured by the median of SRCC and PLCC

[0099] Method CLIVE_50 CLIVE_100 CLIVE_200 KONIQ_50 KONIQ_100 KONIQ_200 TReS 0.67 0.751 0.799 0.713 0.719 0.791 HyperIQA 0.648 0.725 0.79 0.615 0.71 0.776 MetaIQA 0.667 0.718 0.812 0.618 0.620 0.660 DEIQT 0.642 0.769 0.797 0.638 0.682 0.754 MANIQA 0.691 0.769 0.81 0.652 0.755 0.810 LIQE 0.664 0.721 0.733 0.759 0.801 0.832 CLIP 0.664 0.721 0.733 0.736 0.77 0.782 CONTRIQUE 0.695 0.729 0.761 0.733 0.794 0.821 Re-IQA 0.591 0.621 0.701 0.685 0.723 0.754 GRep 0.76 0.791 0.822 0.812 0.836 0.855 The present invention 0.836 0.857 0.875 0.853 0.872 0.883

[0100] Table 2 Performance comparison of synthetic databases under few-shot conditions, measured by the median of SRCC and PLCC

[0101] Method CSIQ_50 CSIQ_100 CSIQ_200 LIVE_50 LIVE_100 LIVE_200 PIPAL50 PIPAL100 PIPAL200 TReS 0.791 0.811 0.878 0.901 0.927 0.957 0.186 0.349 0.501 HyperIQA 0.79 0.824 0.909 0.892 0.912 0.929 0.102 0.302 0.379 MetaIQA 0.821 0.891 0.941 0.92 0.942 0.955 0.332 0.348 0.371 DEIQT 0.794 0.847 0.874 0.909 0.928 0.957 0.396 0.410 0.436 MANIQA 0.838 0.891 0.924 0.904 0.934 0.948 0.136 0.361 0.47 LIQE 0.793 0.89 0.935 0.871 0.906 0.922 - - - CLIP 0.841 0.892 0.941 0.896 0.923 0.941 0.254 0.303 0.368 CONTRIQUE 0.84 0.926 0.94 0.891 0.922 0.943 0.379 0.437 0.488 Re-IQA 0.893 0.907 0.923 0.884 0.894 0.929 0.280 0.350 0.431 GRep 0.878 0.914 0.941 0.926 0.937 0.953 0.390 0,450 0.498 The present invention 0.893 0.917 0.941 0.932 0.943 0.968 0.474 0.512 0.546

[0102] Table 3 Performance comparison of the average value of SRCC and PLCC on the synthetic database

[0103]

[0104] Table 4 Performance comparison of the average value of SRCC and PLCC on the real database

[0105]

[0106] It can be seen from Table 1, Table 2, Table 3 and Table 4 that the method proposed in the present invention is in the leading position on all data sets.

[0107] Based on the same inventive concept, the present application also provides a device corresponding to the method in Example 1, see Example 2 for details.

[0108] Embodiment 2

[0109] In this embodiment, a device for evaluating the quality of a few-sample image based on a visual language model is provided. Figure 4 As shown, including:

[0110] The meta-cue pre-training module is used to divide the support dataset and the query dataset as the meta-knowledge task in the distorted dataset according to the distortion type; in the inner loop process, the features are extracted through the partially learnable visual encoder and the frozen text encoder according to the input support dataset, and the quality score of the distorted image is output according to the visual-text feature similarity, and then the training loss is calculated according to the quality label corresponding to the image to optimize the meta-cue, and the meta-cue includes the quality visual cue and the quality text cue. The quality text cue is used as the pre-trained CLIP model V qua The input of the text encoder is the quality visual cues as the pre-trained CLIP model V qua In the outer loop, the query dataset is used to optimize the meta-prompt according to the inner loop process, and the pre-trained CLIP model V is obtained. qua ;

[0111] The quality-aware gradient regularization module is used to pass images, quality text cues, and semantic cues into the pre-trained CLIP model V qua Image encoder and text encoder and semantic CLIP model V sem , quantify the pre-trained CLIP model V by calculating the KL divergence between zero-shot semantic prediction probabilities quaThe quality optimization direction of the IQA task is obtained by calculating the cross entropy between the model’s prediction and the basic facts. Finally, the IQA task gradient is adjusted by clipping the gradient aligned with the general semantic direction in the shared representation space, and the regularized gradient is generated for backward updating to balance the learning between quality and semantics.

[0112] Since the device introduced in the second embodiment of the present invention is a device used to implement the method of the first embodiment of the present invention, those skilled in the art can understand the specific structure and deformation of the device based on the method introduced in the first embodiment of the present invention, so it is not described here in detail. All devices used in the method of the first embodiment of the present invention belong to the scope of protection of the present invention.

[0113] The present invention introduces Gradient Regulated Meta-Prompted Image Quality Assessment (GRMP-IQA). First, meta-prompt pre-training is used to merge meta-knowledge from various image distortions using a two-layer gradient optimization process, which is used to refine the initialization of text and visual cues, thereby enhancing the adaptability of the CLIP model to IQA tasks and reducing the sensitivity of prompt initialization to various IQA scenarios. Then, quality-aware gradient regularization is used to correct the bias introduced by the visual-language model (CLIP) that over-focuses on semantic content, ensuring a more accurate assessment of image quality, so that the pre-trained CLIP model can adapt to BIQA tasks with a small number of training samples.

[0114] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0115] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0116] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0117] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0118] Although the specific implementation modes of the present invention are described above, those skilled in the art should understand that the specific implementation modes described are only illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A few-sample image quality assessment method based on a visual language model, characterized in that: include: Meta-cue pre-training process: In the distorted dataset, the support dataset and query dataset as the meta-knowledge task are divided according to the distortion type; In the inner loop, features are extracted through a partially learnable visual encoder and a frozen text encoder according to the input support dataset, and the quality score of the distorted image is output according to the visual-text feature similarity. Then, the training loss is calculated according to the quality label corresponding to the image to optimize the meta-cue, which includes quality visual cues and quality text cues. The quality text cues are used as the pre-trained CLIP model V qua The input of the text encoder is the quality visual cues as the pre-trained CLIP model V qua In the outer loop, the query dataset is used to optimize the meta-prompt according to the inner loop process, and the pre-trained CLIP model V is obtained. qua ; Quality-aware gradient regularization process: images, quality text hints, and semantic hints are passed to the pre-trained CLIP model V qua Image encoder and text encoder and semantic CLIP model V sem , quantify the pre-trained CLIP model V by calculating the KL divergence between zero-shot semantic prediction probabilities qua The quality optimization direction of the IQA task is obtained by calculating the cross entropy between the model’s prediction and the basic facts. Finally, the IQA task gradient is adjusted by clipping the gradient aligned with the general semantic direction in the shared representation space, and the regularized gradient is generated for backward updating to balance the learning between quality and semantics.

2. The method according to claim 1, characterized in that: In the meta-prompt pre-training process, the support dataset and the query dataset as the meta-knowledge task are divided in the distorted dataset according to the distortion type, which specifically includes: constructing a K t -way task, denoted as T t , and then use it to construct the meta-training set in, and denote the support dataset and query dataset of each task respectively, and T denotes the total number of tasks. K tasks are randomly sampled from the meta-training set 1≤k≤T as mini-batches to perform two-layer gradient optimization.

3. The method according to claim 1, characterized in that: During the meta-prompt pre-training process, for an input image x, the textual meta-prompt q is processed by the text encoder g(·) i ; At the same time, the image x is encoded by the visual encoder and the [CLS] tag is used to extract the feature vector f; therefore, the probability of predicting high quality is mathematically expressed as: Among them, q high and q low denote high-quality and low-quality category cues, respectively, p(q high |x) represents the probability that the estimated image quality x is high quality, τ is the temperature parameter learned by CLIP, and <,> represents the cosine similarity; Then, the labeled quality scores are rescaled to 0-1, recorded as y, and the loss function is calculated as: L=-(ylog(p(q high |x))+(1-y)log(1-p(q high |x)))。 4. The method according to claim 3, characterized in that: The inner loop process is used to make the meta-hint θ = [θ T ,θ V ]Adaptation support dataset Determine the loss first The model parameters are then updated on the support set using an internal learning rate α, specified as: The outer loop process is used to Adjust model parameters θ i ′: Aggregate the gradients from all tasks to update the final model parameters, following the update rule: Where β represents the external learning rate.

5. The method according to claim 1, characterized in that: The quality visual cue adopts deep cue tuning as the visual meta-cue, and the input embedding of the self-attention module of the l-layer of the vit-based image encoder in CLIP is represented as where f l represents the classification label embedding, Represents image patch embeddings; deep cue fine-tuning will learn the labels The above token sequence added to each ViT layer in the CLIP image encoder; the multi-head attention module of layer l then processes the input tokens as follows: [f l ,_,H l ]=Layer l ([f l1 ,P l-1 ,H l-1 ]) Among them, P l The output embedding of is discarded, denoted as _, and is not fed to subsequent layers.

6. The method according to claim 1, characterized in that: The quality text hint is defined as q i ={v1,v2,…,v M ,[quality]}, where q i Indicates i th The learnable text features of the class, v M Represents a learnable vector, embedding the quality label [quality]∈{high quality,low quality}.

7. The method according to claim 1, characterized in that: The quality-aware gradient regularization process specifically includes: given an input image x and a semantic hint w i = "a photo of a[class].", using the semantic CLIP model and the pre-trained CLIP model to predict the class probability distribution p of image x sem (w i |x) and p qua (w i |x), and calculate the gradient of the KL divergence loss As a measure of general semantic direction; the formula for calculating KL divergence loss is: At the same time, according to the quality text prompt θ T and image x, calculate the quality loss gradient between the predicted quality and the actual quality label As the quality direction; finally, by cutting and general semantic direction G sem Aligned gradients to refine the IQA task direction G qua →G qgr .

8. The method according to claim 7, characterized in that Cropping and general semantic direction G sem The aligned gradients specifically include: The gradient Represented as G qua , Represented as G sem , if G qua With G sem If the angle between the two directions is less than 90 degrees, then the sem Parallel components G || Shear G qua , if G qua With G sem If the angle between the two is greater than or equal to 90 degrees, the mass gradient G is not adjusted. qua ; The regularization strategy is expressed as follows: Among them, λ=1 means that G qua Projection to G sem in the orthogonal direction.

9. A device for evaluating image quality based on a few samples using a visual language model, characterized in that: include: The meta-cue pre-training module is used to divide the support dataset and query dataset as meta-knowledge tasks in the distorted dataset according to the distortion type; In the inner loop, features are extracted through a partially learnable visual encoder and a frozen text encoder according to the input support dataset, and the quality score of the distorted image is output according to the visual-text feature similarity. Then, the training loss is calculated according to the quality label corresponding to the image to optimize the meta-cue, which includes quality visual cues and quality text cues. The quality text cues are used as the pre-trained CLIP model V qua The input of the text encoder is the quality visual cues as the pre-trained CLIP model V qua In the outer loop, the query dataset is used to optimize the meta-prompt according to the inner loop process, and the pre-trained CLIP model V is obtained. qua ; The quality-aware gradient regularization module is used to pass images, quality text cues, and semantic cues into the pre-trained CLIP model V qua Image encoder and text encoder and semantic CLIP model V sem , quantify the pre-trained CLIP model V by calculating the KL divergence between zero-shot semantic prediction probabilities qua The quality optimization direction of the IQA task is obtained by calculating the cross entropy between the model’s prediction and the basic facts. Finally, the IQA task gradient is adjusted by clipping the gradient aligned with the general semantic direction in the shared representation space, and the regularized gradient is generated for backward updating to balance the learning between quality and semantics.

Images