A few-shot image quality assessment method and device based on a visual language model

Abstract

The application discloses a few-shot image quality assessment method and device based on a visual language model, and relates to the technical field of image quality assessment.The application introduces a gradient regulation meta prompt image quality assessment (GRMP-IQA), first adopts a double-layer gradient optimization process to combine meta knowledge from various image distortions through meta prompt pre-training, thereby being used for refining the initialization of text and visual prompts, enhancing the adaptability of the CLIP model to the IQA task, and reducing the sensitivity of prompt initialization to various IQA scenes; then, the quality perception gradient regularization corrects the deviation introduced by the visual-language model (CLIP) paying excessive attention to semantic content, ensures more accurate assessment of the image quality, and makes the pre-trained CLIP model adapt to the BIQA task with a small number of training samples.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image quality assessment, and particularly relates to a few-shot image quality assessment method and device based on a visual language model. BACKGROUND

[0002] With the advent of the mobile internet era, the focus on computer vision has shifted from the initial compression and image processing to handling user-generated content, such as smartphone photos and videos, and recently, artificial intelligence-generated content. This progress has accelerated a significant escalation in the demand for effective image quality assessment (IQA) techniques, emphasizing the importance of developing methods that can proficiently assess image quality without reference images. In the field of image quality assessment, methods are roughly divided into: full-reference IQA (FR-IQA) with complete reference images, reduced-reference IQA (RR-IQA) with partial reference data, and blind IQA (BIQA) without any reference images. BIQA presents a unique and challenging problem in practical applications due to the lack of reference images.

[0003] Recent BIQA methods attempt to alleviate this problem, DEIQT introduces a transformer decoder to help the visual transformer (ViT) adapt to the IQA task. In addition, some self-supervised methods utilize contrastive learning during pre-training to extract general quality features, which can improve performance even with limited label availability. Data-driven BIQA models based on deep neural networks have made significant progress in recent years. However, the quality score of distorted images is usually measured using the mean opinion score (MOS), which is the average of multiple ratings (sometimes up to 120). However, these methods usually require a large pre-training dataset and involve training large-scale neural networks with millions of parameters, so obtaining a sufficient number of IQA training samples is a very time-consuming and laborious task, resulting in a large training cost.

[0004] Recently, pre-trained visual language models (CLIP) have achieved surprising performance in various downstream tasks. In particular, CLIP-IQA shows promising zero-shot generalization capabilities by simply filling in handcrafted prompts (e.g., "good photo") as input to the CLIP encoder in the IQA task. Despite these advances, such models can encounter a domain gap between the upstream training data and the specific requirements of the IQA downstream task, which limits the performance of zero-shot methods. Therefore, data-efficient IQA is expected to provide a more robust and effective approach to predicting image quality compared to zero-shot methods, but remains relatively unexplored. SUMMARY

[0005] The technical problem to be solved by the present invention is to provide a method and apparatus for few-sample image quality assessment based on a visual language model. By introducing gradient modulated meta-cue image quality assessment (GRMP-IQA), the pre-trained visual-language model (CLIP) is adapted to the BIQA task with a small number of training samples.

[0006] In a first aspect, the present invention provides a few-shot image quality assessment method based on a visual language model, comprising:

[0007] Meta-hint pre-training process: The distortion dataset is divided into a support dataset and a query dataset based on distortion type, serving as the meta-knowledge task. In the inner loop, features are extracted from the input support dataset using a partially learnable visual encoder and a frozen text encoder, respectively. The quality score of the distortion image is output based on the visual-text feature similarity. Then, the training loss is calculated based on the image's corresponding quality label to optimize the meta-hint. The meta-hint includes quality visual hints and quality text hints, with the quality text hint serving as the pre-trained CLIP model V. qua The text encoder takes as input, and quality visual cues are used as pre-trained CLIP model V. qua The image encoder parameters are part of the algorithm; during the outer loop, the meta-cues are optimized using the query dataset following the inner loop process to obtain the pre-trained CLIP model V. qua ;

[0008] Quality-aware gradient regularization process: The image, quality text cues, and semantic cues are fed into the pre-trained CLIP model V respectively. qua Image encoders and text encoders, as well as the semantic CLIP model V sem The pre-trained CLIP model V is quantified by calculating the KL divergence between zero-sample semantic prediction probabilities. qua The semantic attention level in the model is obtained by calculating the cross-entropy between the model's predictions and the basic facts to obtain the quality optimization direction of the IQA task. Finally, the gradient of the IQA task is adjusted by pruning gradients aligned with the general semantic direction in the shared representation space to generate regularized gradients for backward updates, thus balancing the learning between quality and semantics.

[0009] Secondly, the present invention provides a few-sample image quality assessment device based on a visual language model, comprising:

[0010] The meta-hint pre-training module is used to divide the distorted dataset into a support dataset and a query dataset for the meta-knowledge task based on the distortion type. During the inner loop, features are extracted from the input support dataset using a partially learnable visual encoder and a frozen text encoder, respectively. The quality score of the distorted image is output based on the visual-text feature similarity. Then, the training loss is calculated based on the quality label corresponding to the image to optimize the meta-hint. The meta-hint includes quality visual hints and quality text hints, with the quality text hint serving as the pre-trained CLIP model V. qua The text encoder takes as input, and quality visual cues are used as pre-trained CLIP model V. qua The image encoder parameters are part of the algorithm; during the outer loop, the meta-cues are optimized using the query dataset following the inner loop process to obtain the pre-trained CLIP model V. qua ;

[0011] The quality-aware gradient regularization module is used to input image, quality text cues, and semantic cues into the pre-trained CLIP model V, respectively. qua Image encoders and text encoders, as well as the semantic CLIP model V sem The pre-trained CLIP model V is quantified by calculating the KL divergence between zero-sample semantic prediction probabilities. qua The semantic attention level in the model is obtained by calculating the cross-entropy between the model's predictions and the basic facts to obtain the quality optimization direction of the IQA task. Finally, the gradient of the IQA task is adjusted by pruning gradients aligned with the general semantic direction in the shared representation space to generate regularized gradients for backward updates, thus balancing the learning between quality and semantics.

[0012] The technical solution provided by this invention has at least the following technical effects:

[0013] By introducing Gradient Adjusted Meta-Cue Image Quality Assessment (GRMP-IQA), meta-knowledge from various image distortions is first incorporated through a two-layer gradient optimization process during meta-cue pre-training. This refines the initialization of textual and visual cues, enhancing the CLIP model's adaptability to IQA tasks and reducing the sensitivity of cue initialization to various IQA scenarios. Then, quality-aware gradient regularization corrects the bias introduced by the Visual-Language Model (CLIP)'s overemphasis on semantic content, ensuring a more accurate assessment of image quality. This allows the pre-trained CLIP model to adapt to BIQA tasks with limited training samples. Attached Figure Description

[0014] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0015] Figure 1This is a schematic diagram of the overall process of the method in Embodiment 1 of the present invention;

[0016] Figure 2 This is a schematic diagram illustrating the principle of gradient adjustment element prompting image quality assessment in Embodiment 1 of the present invention;

[0017] Figure 3 This is a schematic diagram comparing the gMAD evaluation results of DEIQT and GRMP-IQA of the present invention in Embodiment 2 of the present invention;

[0018] Figure 4 This is a schematic diagram of the device in Embodiment 2 of the present invention. Detailed Implementation

[0019] This invention provides a method and apparatus for few-sample image quality assessment based on a visual language model. By introducing Gradient Adjusted Meta-Cue Image Quality Assessment (GRMP-IQA), the pre-trained visual-language model (CLIP) is adapted to the BIQA task with a small number of training samples.

[0020] The overall concept of the technical solutions in the embodiments of the present invention is as follows:

[0021] Gradient-modulated meta-cue image quality assessment (GRMP-IQA) was introduced, such as Figure 2 As shown, GRMP-IQA consists of two main components: the Meta-Cue Pre-training Module (MPP) and Quality-Aware Gradient Regularization (QGR). That is, this invention includes a meta-cue training phase and a quality-aware gradient regularization phase. Specifically, the meta-cue training phase includes two parts: 1. Using the support dataset D in the inner loop... support Optimize; 2. Use the query dataset D in the outer loop. query Optimization is then performed. To this end, the existing large-scale distortion dataset is first divided into support datasets D based on distortion type, serving as the meta-knowledge task. support And query dataset D query This is then passed into the inner and outer loops respectively. In the inner loop, the invention is based on D... support The input image is processed by extracting features from a partially learnable visual encoder and a frozen text encoder, respectively. Based on the visual-text feature similarity, a quality score for the distorted image is output. The training loss is then calculated based on the image's corresponding quality label to optimize the learnable meta-cues (including quality visual cues and quality text cues) in this invention. During the outer loop, based on the updated meta-cues from the inner loop, the query dataset D is used again. queryThe meta-cues are trained following an inner loop process. This process effectively simulates the model's generalization across different scenarios and effectively learns shared quality-aware priors across various scenarios. In the quality-aware gradient regularization phase, images, semantic cues, and learnable quality text cues are fed into the image encoder (where the quality visual cues obtained in the first stage of training are used as part of the image encoder parameters), text encoder, and semantic CLIP, respectively, to calculate CE loss and KL loss, representing the IQA task direction and the general semantic direction, respectively. During gradient regularization, the IQA task direction is refined by cropping gradients aligned with the general semantic direction, ensuring that the model prioritizes image quality and achieves accurate image quality assessment. The meta-cue training phase (MPP) focuses on pre-training the visual text cues, utilizing shared meta-knowledge about distortion to facilitate rapid adaptation to various IQA tasks. The quality-aware gradient regularization phase (QGR) plays a crucial role in fine-tuning by adjusting the update gradients to mitigate overfitting to semantic content. The specific stages are shown below.

[0022] Meta-hint training phase: A small batch of samples is randomly selected from the meta-knowledge task and divided into support sets D. support and query set D query A two-layer gradient descent method from D support The internal circulation on top is pushed to D. query The outer loop is used to optimize learnable meta-cues [θ] T ,θ V ], where θ T The quality text prompt, θ V These visual cues represent quality and are then used as initial conditions for the fine-tuning phase.

[0023] Quality-aware gradient regularization stage: Given an input image x and semantic cue w i = "a photo of a [class].", using the original semantic CLIP model and a pre-trained CLIP model to predict the class probability distribution p of image x. sem (w i |x) and p qua (w i |x), and calculate the gradient of the KL divergence loss. As a measure of general semantic direction. Simultaneously, based on quality text cues θ T Given image x, calculate the quality loss gradient between the predicted quality and the actual quality label. As a quality direction. Finally, through pruning and general semantic direction G. sem Aligned gradients to refine the IQA task direction G qua →G qgrThis approach ensures that the model prioritizes image quality, thereby minimizing the impact of semantic noise.

[0024] 1. Meta hints

[0025] In this embodiment, a meta-cue is proposed, comprising two learnable cues: a quality visual cue and a quality text cue. The quality text cue serves as input to the text encoder, while the quality visual cue is included as part of the image encoder parameters. The meta-cue combines adjustments to the text and visual cues, working in a mutually reinforcing manner. The goal is to enable these learnable cues to gather shared knowledge across different distortion scenarios, thereby facilitating CLIP's rapid adaptation to the IQA domain.

[0026] 1.1 Visual Quality Cueing

[0027] Depth cue tuning (DPT) is employed as the visual meta-cue. Formally, the input embedding from the self-attention module of layer l of the vit-based image encoder in CLIP is represented as follows: Where f l Indicates category tag embedding, Represents image patch embedding; depth cue tuning will enable learnable tags. The above-mentioned tag sequence is added to each ViT layer in the CLIP image encoder; the multi-head attention module of layer l then processes the input tags as follows:

[0028]

[0029] Among them, P l The output embedding is discarded, represented as -, and is not fed into subsequent layers.

[0030] 1.2 Quality Text Prompts

[0031] Text meta-hints are optimized by modeling continuous vectors of context words, achieving end-to-end learning while pre-training parameters are frozen. Specifically, quality text hints are defined as q... i ={v1,v2,…,v M ,[quality]}, where q i Indicate i th Learnable text features of the class, v M This represents a learnable vector, with embedded quality labels [quality] ∈ {high quality, low quality}. In this embodiment of GRMP-IQA, the vector length is set to 4.

[0032] 2. Meta-cue pre-training

[0033] As mentioned earlier, in the pre-training phase of cue optimization, the focus is solely on optimizing the learnable meta-cues, defined as θ = [θ T ,θ V ]. Wherein, θ V For quality visual cues, θ T For quality text cues, in this case, for the input image x, the text meta-cue q is processed by the text encoder g(·). i Simultaneously, image x is encoded by a visual encoder, and the feature vector f is extracted using [CLS] labeling; therefore, the probability of predicting high quality is mathematically expressed as:

[0034]

[0035] Where, q high and q low p(q) represents category hints for high quality and low quality, respectively. high |x) represents the probability that the estimated image quality x is high quality, τ is the temperature parameter learned by CLIP, and <,> represents the cosine similarity;

[0036] Then, the labeled quality scores are rescaled to 0-1, denoted as y, and the loss function is calculated as follows:

[0037] L=-(ylog(p(q high |x))+(1-y)log(1-p(q high |x))) (3).

[0038] 2.1 Constructing Distorted Meta-Knowledge Task

[0039] The ability to detect various types of image distortion is crucial for developing BIQA models with strong generalization capabilities across different scenarios. Furthermore, the effectiveness of cue tuning largely depends on the initial cue configuration. This initial setup significantly impacts the CLIP visual-language model's ability to quickly adapt to different IQA scenarios. Inspired by the "learning-to-learn" spirit inherent in the deep meta-learning paradigm, this invention proposes an optimization-based method for effectively pre-training visual-text cues. These cues incorporate shared quality insights from various image distortions, thereby enhancing CLIP's rapid adaptability to IQA tasks. To investigate the general laws governing image distortion, a model targeting K is first constructed... t The BIQA task of the -way task, denoted as T t Then use it to construct the meta-training set. in, and Let represent the supporting dataset and query dataset for each task, respectively, and T represent the total number of tasks. To simulate the fast generalization process for different distortions in a BIQA scenario, K tasks are randomly sampled from the meta-training set 1≤k≤T as mini-batches to perform bi-layer gradient optimization.

[0040] 2.2 Distortion-perceptual meta-cue learning

[0041] This method employs a two-layer gradient descent technique to connect the learning process from the support set to the query set, thereby facilitating the acquisition of quality prior knowledge through prompts. Specifically, it consists of two optimization steps. In the inner loop (first layer), the gradient of the prompt parameters is computed using the support set, and the first update is applied. In the outer loop (second layer), the performance of the updated model on the query set is evaluated. This two-layer structure enables the prompts to quickly generalize across different BIQA contexts by optimizing the meta-clips for each unique distortion scenario.

[0042] 2.2.1 Internal Circulation Process

[0043] The goal is to make the meta-hint θ = [θ T ,θ V Adaptation Support Dataset During the first-level update, determine the loss. The model parameters are then updated on the support set using the internal learning rate α, specified as:

[0044]

[0045] 2.2.2 External Circulation Process

[0046] Similarly, the second-level update is based on the query set. Adjusting model parameters θ i ′:

[0047]

[0048] The gradients from all tasks are aggregated to update the final model parameters, following the update rules:

[0049]

[0050] Where β represents the external learning rate.

[0051] This method of selecting and optimizing tasks from the meta-training set effectively trains the learnable meta-cues θ = [θ]. T ,θ V This ensures their generalization across various image distortions.

[0052] 3. Quality-perceived gradient regularization

[0053] The original CLIP model was pre-trained on a classification task to obtain a semantic CLIP model, denoted as V. sem The CLIP model pre-trained in the meta-cue pre-training module uses V qua express.

[0054] Recent research has highlighted the inconsistency between the low-level tasks of IQA and the high-level semantic representations from various upstream tasks. In this context, models may overemphasize semantic content at the expense of image quality, leading to overfitting and decreased generalization ability. To mitigate this problem, this embodiment proposes a Quality-Aware Gradient Regularization (QGR) technique. This method utilizes a semantic CLIP model V sem Directional guidance for zero-shot semantic prediction in the pre-trained CLIP model V qua The quality gradient update is refined during fine-tuning of the IQA task. The goal is to reduce the overemphasis on semantics and promote a balanced assessment of image quality.

[0055] Specifically, in order to obtain the direction G of general semantic knowledge sem First, a hard prompt w was designed. i =aphoto ofa[class], where [class] belongs to one of nine semantic categories: {"animals", "cityscape", "people", "indoors", "landscape", "night", "plants", "still life", "others"}. The prompt w i Input into CLIP model V sem and V qua Extract text features from the text encoder and combine them with model V. sem and V qua The image features extracted by the visual encoder are aligned to obtain the zero-sample semantic prediction probability p. sem (w i |x) and p qua (w i |x). For the pre-trained CLIP model V qua The goal of fine-tuning is not to overemphasize semantic information. Therefore, by calculating p... sem (w i |x) and p qua (w i The Kullback-Leibler (KL) divergence between |x) is used to quantify the fine-tuning of the CLIP model V. qua Semantic attention level in:

[0056]

[0057] Conversely, by calculating the model's prediction p(q) high The cross-entropy L between |x) and the basic fact yce (V qua This leads to quality optimization directions in the IQA domain, as described in Equation 3. The learning of quality and semantics is balanced by adjusting the task's gradient within a shared representation space.

[0058] Specifically, the gradient Represented as G qua , Represented as G sem The relationship between them can be described in two aspects:

[0059] (1) If G qua With G sem If the angle between the two is less than 90 degrees, it indicates that the optimization direction of the small-sample IQA quality knowledge is consistent with that of the general semantic knowledge. In this case, along its semantic direction G... sem Parallel component G || Cut G qua This is done by adjusting the model's original quality-aware optimization path to prevent overfitting to semantic relevance.

[0060] (2) If G qua With G sem If the included angle is greater than or equal to 90 degrees, it indicates that they are in different directions during the improvement process. In this case, the mass gradient G is not adjusted. qua This allows the model to learn quality-aware features based on the original optimization direction.

[0061] The regularization strategy is mathematically represented as follows:

[0062]

[0063] The introduction of λ to generalize the formula provides flexibility in terms of the degree of guidance from general knowledge in control applications. Specifically, λ = 1 indicates that G... qua Projected to G sem In the orthogonal direction.

[0064] The contributions of the technical solution in this embodiment are summarized as follows:

[0065] (1) A meta-cue pre-training module is proposed, which organizes meta-tasks by image distortion type and optimizes soft cues through bi-layer gradient descent. This method allows soft cues to acquire shared meta-knowledge across different distortions, greatly improving the model's rapid adaptability to a wide range of IQA scenarios.

[0066] (2) A novel quality-aware gradient regularization method is proposed to adjust the update gradient during fine-tuning, maintaining a balance between semantic and quality information. This method prunes the gradients that are consistent with the semantic direction, ensuring that the model prioritizes image quality while integrating semantic context where relevant.

[0067] (3) Furthermore, it facilitates the integration of text and visual cue optimization in a collaborative manner, thereby enhancing their mutual effectiveness and providing superior generalization under data-efficient settings.

[0068] Example 1

[0069] This embodiment provides a few-shot image quality assessment method based on a visual language model, such as... Figure 1 As shown, it includes:

[0070] Meta-hint pre-training process: The distortion dataset is divided into a support dataset and a query dataset based on distortion type, serving as the meta-knowledge task. In the inner loop, features are extracted from the input support dataset using a partially learnable visual encoder and a frozen text encoder, respectively. The quality score of the distortion image is output based on the visual-text feature similarity. Then, the training loss is calculated based on the image's corresponding quality label to optimize the meta-hint. The meta-hint includes quality visual hints and quality text hints, with the quality text hint serving as the pre-trained CLIP model V. qua The text encoder takes as input, and quality visual cues are used as pre-trained CLIP model V. qua The image encoder parameters are part of the algorithm; during the outer loop, the meta-cues are optimized using the query dataset following the inner loop process to obtain the pre-trained CLIP model V. qua ;

[0071] Quality-aware gradient regularization process: The image, quality text cues, and semantic cues are fed into the pre-trained CLIP model V respectively. qua Image encoders and text encoders, as well as the semantic CLIP model V sem The pre-trained CLIP model V is quantified by calculating the KL divergence between zero-sample semantic prediction probabilities. qua The semantic attention level in the model is obtained by calculating the cross-entropy between the model's predictions and the basic facts to obtain the quality optimization direction of the IQA task. Finally, the gradient of the IQA task is adjusted by pruning gradients aligned with the general semantic direction in the shared representation space to generate regularized gradients for backward updates, thus balancing the learning between quality and semantics.

[0072] In one possible implementation, during the meta-hint pre-training process, the distortion dataset is divided into a support dataset and a query dataset for the meta-knowledge task based on the distortion type. Specifically, this includes constructing a dataset for K... t The BIQA task of the -way task, denoted as T t Then use it to construct the meta-training set. in, and Let T represent the supporting dataset and query dataset for each task, respectively, and T represent the total number of tasks. K tasks are randomly sampled from the meta-training set 1≤k≤T as mini-batches to perform bi-layer gradient optimization.

[0073] During the meta-hint pre-training process, for the input image x, the text meta-hint q is processed by the text encoder g(·). i Simultaneously, image x is encoded by a visual encoder, and the feature vector f is extracted using [CLS] labeling; therefore, the probability of predicting high quality is mathematically expressed as:

[0074]

[0075] Where, q high and q low p(q) represents category hints for high quality and low quality, respectively. high |x) represents the probability that the estimated image quality x is high quality, τ is the temperature parameter learned by CLIP, and <,> represents the cosine similarity;

[0076] Then, the labeled quality scores are rescaled to 0-1, denoted as y, and the loss function is calculated as follows:

[0077] L=-(ylog(p(q high |x))+(1-y)log(1-p(q high |x))).

[0078] The inner loop process is used to make the meta-hint θ = [θ T ,θ V Adaptation Support Dataset First determine the loss The model parameters are then updated on the support set using the internal learning rate α, specified as:

[0079]

[0080] The outer loop process is used to process the query set. Adjusting model parameters θ i ′:

[0081]

[0082] The gradients from all tasks are aggregated to update the final model parameters, following the update rules:

[0083]

[0084] Where β represents the external learning rate.

[0085] The quality visual cues employ depth cue tuning as visual meta-cues, embedding the input from the self-attention module of layer l of the vit-based image encoder in CLIP as follows: Where f l Indicates category tag embedding, Represents image patch embedding; depth cue tuning will enable learnable tags. The above-mentioned tag sequence is added to each ViT layer in the CLIP image encoder; the multi-head attention module of layer l then processes the input tags as follows:

[0086]

[0087] Among them, P l The output embedding is discarded, represented as -, and is not fed into subsequent layers;

[0088] The quality text prompt is defined as q. i ={v1,v2,...,v M ,[quality]}, where q i Indicate i th Learnable text features of the class, v M Represents a learnable vector, with embedded quality labels [quality] ∈ {high quality, low quality}.

[0089] The quality-aware gradient regularization process specifically includes: given an input image x and semantic cue w i = "aphoto of a[class].", using a semantic CLIP model and a pre-trained CLIP model to predict the class probability distribution p of image x. sem (w i |x) and p qua (w i |x), and calculate the gradient ▽L of the KL divergence loss. kl As a measure of general semantic direction, the formula for calculating the KL divergence loss is:

[0090]

[0091] Meanwhile, according to the quality text prompt θ T Given image x, calculate the quality loss gradient between the predicted quality and the actual quality label. As a quality direction; finally, through pruning and general semantic direction G sem Aligned gradients to refine the IQA task direction G qua →G qgr .

[0092] Clipping and general semantic direction G semThe alignment gradient specifically includes:

[0093] gradient Represented as G qua , Represented as G sem If G qua With G sem If the included angle is less than 90 degrees, then it follows the semantic direction G. sem Parallel component G || Cut G qua If G qua With G sem If the included angle is greater than or equal to 90 degrees, then the mass gradient G is not adjusted. qua The regularization strategy is expressed as follows:

[0094]

[0095] Where λ = 1 indicates that G qua Projected to G sem In the orthogonal direction.

[0096] Figure 3 This diagram illustrates a comparison of the gMAD evaluation results of DEIQT and the GRMP-IQA of this invention. (a) DEIQT is fixed at a low quality level. (b) DEIQT is fixed at a high quality level. (c) This invention is fixed at a low quality level. (d) This invention is fixed at a high quality level.

[0097] Extensive experimental results highlight the effectiveness and efficiency of the method presented in this embodiment. Notably, using only 200 data samples, the proposed method outperforms state-of-the-art models on the CLIIVE dataset (20% of the training data) and achieves highly competitive performance on the KonIQ dataset (2% of the training data), as shown in Tables 1-4. Here, CLIIVE and KonIQ are the dataset names, and the number of images used for training is indicated by the underlined text. The first column lists representative image quality assessment methods, such as TReS, which uses a hybrid architecture with convolutional neural networks and Transformer networks for image quality prediction. Bold data entries represent the best results.

[0098] Table 1 compares the performance of real-world databases under limited sample conditions, measured by the median of SRCC and PLCC.

[0099] Method CLIVE_50 CLIVE_100 CLIVE_200 KONIQ_50 KONIQ_100 KONIQ_200 TReS 0.67 0.751 0.799 0.713 0.719 0.791 HyperIQA 0.648 0.725 0.79 0.615 0.71 0.776 MetaIQA 0.667 0.718 0.812 0.618 0.620 0.660 DEIQT 0.642 0.769 0.797 0.638 0.682 0.754 MANIQA 0.691 0.769 0.81 0.652 0.755 0.810 LIQE 0.664 0.721 0.733 0.759 0.801 0.832 CLIP 0.664 0.721 0.733 0.736 0.77 0.782 CONTRIQUE 0.695 0.729 0.761 0.733 0.794 0.821 Re-IQA 0.591 0.621 0.701 0.685 0.723 0.754 GRepQ 0.76 0.791 0.822 0.812 0.836 0.855 The invention 0.836 0.857 0.875 0.853 0.872 0.883

[0100] Table 2 compares the performance of synthetic databases under few-sample conditions, measured by the median of SRCC and PLCC.

[0101] Method CSIQ_50 CSIQ_100 CSIQ_200 LIVE_50 LIVE_100 LIVE_200 PIPAL50 PIPAL100 PIPAL200 TReS 0.791 0.811 0.878 0.901 0.927 0.957 0.186 0.349 0.501 HyperIQA 0.79 0.824 0.909 0.892 0.912 0.929 0.102 0.302 0.379 MetaIQA 0.821 0.891 0.941 0.92 0.942 0.955 0.332 0.348 0.371 DEIQT 0.794 0.847 0.874 0.909 0.928 0.957 0.396 0.410 0.436 MANIQA 0.838 0.891 0.924 0.904 0.934 0.948 0.136 0.361 0.47 LIQE 0.793 0.89 0.935 0.871 0.906 0.922 - - - CLIP 0.841 0.892 0.941 0.896 0.923 0.941 0.254 0.303 0.368 CONTRIQUE 0.84 0.926 0.94 0.891 0.922 0.943 0.379 0.437 0.488 Re-IQA 0.893 0.907 0.923 0.884 0.894 0.929 0.280 0.350 0.431 GRepQ 0.878 0.914 0.941 0.926 0.937 0.953 0.390 0,450 0.498 The invention 0.893 0.917 0.941 0.932 0.943 0.968 0.474 0.512 0.546

[0102] Table 3 compares the performance of SRCC and PLCC averages in the synthetic database.

[0103]

[0104] Table 4 compares the performance of SRCC and PLCC using average values ​​in a real database.

[0105]

[0106] As can be seen from Tables 1, 2, 3, and 4, the method proposed in this invention is leading on all datasets.

[0107] Based on the same inventive concept, this application also provides an apparatus corresponding to the method in Embodiment 1, as detailed in Embodiment 2.

[0108] Example 2

[0109] This embodiment provides a few-sample image quality assessment device based on a visual language model, such as... Figure 4 As shown, it includes:

[0110] The meta-hint pre-training module is used to divide the distorted dataset into a support dataset and a query dataset for the meta-knowledge task based on the distortion type. During the inner loop, features are extracted from the input support dataset using a partially learnable visual encoder and a frozen text encoder, respectively. The quality score of the distorted image is output based on the visual-text feature similarity. Then, the training loss is calculated based on the quality label corresponding to the image to optimize the meta-hint. The meta-hint includes quality visual hints and quality text hints, with the quality text hint serving as the pre-trained CLIP model V. qua The text encoder takes as input, and quality visual cues are used as pre-trained CLIP model V. qua The image encoder parameters are part of the algorithm; during the outer loop, the meta-cues are optimized using the query dataset following the inner loop process to obtain the pre-trained CLIP model V. qua ;

[0111] The quality-aware gradient regularization module is used to input image, quality text cues, and semantic cues into the pre-trained CLIP model V, respectively. qua Image encoders and text encoders, as well as the semantic CLIP model V sem The pre-trained CLIP model V is quantified by calculating the KL divergence between zero-sample semantic prediction probabilities. quaThe semantic attention level in the model is obtained by calculating the cross-entropy between the model's predictions and the basic facts to obtain the quality optimization direction of the IQA task. Finally, the gradient of the IQA task is adjusted by pruning gradients aligned with the general semantic direction in the shared representation space to generate regularized gradients for backward updates, thus balancing the learning between quality and semantics.

[0112] Since the apparatus described in Embodiment 2 of the present invention is an apparatus used to implement the method of Embodiment 1 of the present invention, those skilled in the art can understand the specific structure and variations of the apparatus based on the method described in Embodiment 1 of the present invention, and therefore will not be described again here. All apparatuses used in the method of Embodiment 1 of the present invention fall within the scope of protection of the present invention.

[0113] This invention introduces Gradient Adjusted Meta-Cue Image Quality Assessment (GRMP-IQA). First, meta-cue pre-training employs a two-layer gradient optimization process to incorporate meta-knowledge from various image distortions, thereby refining the initialization of textual and visual cues. This enhances the CLIP model's adaptability to IQA tasks and reduces the sensitivity of cue initialization to various IQA scenarios. Then, quality-aware gradient regularization corrects the bias introduced by the Visual-Language Model (CLIP)'s overemphasis on semantic content, ensuring a more accurate assessment of image quality. This allows the pre-trained CLIP model to adapt to BIQA tasks with limited training samples.

[0114] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0115] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0116] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0117] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0118] While specific embodiments of the present invention have been described above, those skilled in the art should understand that the specific embodiments described are merely illustrative and not intended to limit the scope of the invention. Equivalent modifications and variations made by those skilled in the art in accordance with the spirit of the invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for assessing the quality of few-sample images based on a visual language model, characterized in that, include: Meta-hint pre-training process: In the distorted dataset, support datasets and query datasets are divided according to the distortion type to serve as meta-knowledge tasks; During the inner loop, features are extracted from the input support dataset using a partially learnable visual encoder and a frozen text encoder, respectively. The quality score of the distorted image is then output based on the visual-text feature similarity. The training loss is then calculated based on the image's corresponding quality label to optimize the meta-cues, which include quality visual cues and quality text cues. The quality text cues serve as the pre-trained CLIP model V. qua The text encoder takes as input, and quality visual cues are used as pre-trained CLIP model V. qua The image encoder parameters are part of the algorithm; during the outer loop, the meta-cues are optimized using the query dataset following the inner loop process to obtain the pre-trained CLIP model V. qua ; Quality-aware gradient regularization process: The image, quality text cues, and semantic cues are fed into the pre-trained CLIP model V respectively. qua Image encoders and text encoders, as well as the semantic CLIP model V sem The pre-trained CLIP model V is quantified by calculating the KL divergence between zero-sample semantic prediction probabilities. qua The semantic attention level in the model is obtained by calculating the cross-entropy between the model's predictions and the basic facts to obtain the quality optimization direction of the IQA task. Finally, the gradient of the IQA task is adjusted by pruning gradients aligned with the general semantic direction in the shared representation space to generate regularized gradients for backward updates, thus balancing the learning between quality and semantics.

2. The method according to claim 1, characterized in that: During the meta-hint pre-training process, the distortion dataset is divided into a support dataset and a query dataset for the meta-knowledge task based on the distortion type. Specifically, this includes constructing a dataset for K... t The BIQA task of the -way task, denoted as T t Then use it to construct the meta-training set. in, and Let T represent the supporting dataset and query dataset for each task, respectively, and T represent the total number of tasks. K tasks are randomly sampled from the meta-training set 1≤k≤T as mini-batches to perform bi-layer gradient optimization.

3. The method according to claim 1, characterized in that: During the meta-hint pre-training process, for the input image x, the text meta-hint q is processed by the text encoder g(·). i Simultaneously, image x is encoded by a visual encoder, and the feature vector f is extracted using [CLS] labeling; therefore, the probability of predicting high quality is mathematically expressed as: Where, q high and q low p(q) represents category hints for high quality and low quality, respectively. high |x) represents the probability that the estimated image quality x is high quality, τ is the temperature parameter learned by CLIP, and <,> represents the cosine similarity; Then, the labeled quality scores are rescaled to 0-1, denoted as y, and the loss function is calculated as follows: L=-(ylog(p(q high |x))+(1-y)log(1-p(q high |x)))。 4. The method according to claim 3, characterized in that: The inner loop process is used to make the meta-hint θ = [θ T ,θ V Adaptation Support Dataset First determine the loss The model parameters are then updated on the support set using the internal learning rate α, specified as: The outer loop process is used to process the query set. Adjusting model parameters θ i ′: The gradients from all tasks are aggregated to update the final model parameters, following the update rules: Where β represents the external learning rate.

5. The method according to claim 1, characterized in that: The quality visual cues employ depth cue tuning as visual meta-cues, embedding the input from the self-attention module of layer l of the vit-based image encoder in CLIP as follows: Where f l Indicates category tag embedding, Represents image patch embedding; depth cue tuning will enable learnable tags. The above-mentioned tag sequence is added to each ViT layer in the CLIP image encoder; the multi-head attention module of layer l then processes the input tags as follows: [f l ,_,H l ]=Layer l ([f l1 ,P l-1 ,H l-1 ]) Among them, P l The output embedding is discarded, represented as _, and is not fed into subsequent layers.

6. The method according to claim 1, characterized in that: The quality text prompt is defined as q. i ={v1,v2,…,v M ,[quality]}, where q i Indicate i th Learnable text features of the class, v M Represents a learnable vector, with embedded quality labels [quality] ∈ {high quality, low quality}.

7. The method according to claim 1, characterized in that: The quality-aware gradient regularization process specifically includes: given an input image x and semantic cue w i = "a photo of a [class].", using the semantic CLIP model and a pre-trained CLIP model to predict the class probability distribution p of image x. sem (w i |x) and p qua (w i |x), and calculate the gradient of the KL divergence loss. As a measure of general semantic direction, the formula for calculating the KL divergence loss is: Meanwhile, according to the quality text prompt θ T Given image x, calculate the quality loss gradient between the predicted quality and the actual quality label. As a quality direction; finally, through pruning and general semantic direction G sem Aligned gradients to refine the IQA task direction G qua →G qgr .

8. The method according to claim 7, characterized in that, Clipping and general semantic direction G sem The alignment gradient specifically includes: gradient Represented as G qua , Represented as G sem If G qua With G sem If the included angle is less than 90 degrees, then it follows the semantic direction G. sem Parallel component G || Cut G qua If G qua With G sem If the included angle is greater than or equal to 90 degrees, then the mass gradient G is not adjusted. qua The regularization strategy is expressed as follows: Where λ = 1 indicates that G qua Projected to G sem In the orthogonal direction.

9. A few-sample image quality assessment device based on a visual language model, characterized in that, include: The meta-hint pre-training module is used to divide the distorted dataset into a support dataset and a query dataset for meta-knowledge tasks based on the distortion type. During the inner loop, features are extracted from the input support dataset using a partially learnable visual encoder and a frozen text encoder, respectively. The quality score of the distorted image is then output based on the visual-text feature similarity. The training loss is then calculated based on the image's corresponding quality label to optimize the meta-cues, which include quality visual cues and quality text cues. The quality text cues serve as the pre-trained CLIP model V. qua The text encoder takes as input, and quality visual cues are used as pre-trained CLIP model V. qua The image encoder parameters are part of the algorithm; during the outer loop, the meta-cues are optimized using the query dataset following the inner loop process to obtain the pre-trained CLIP model V. qua ; The quality-aware gradient regularization module is used to input image, quality text cues, and semantic cues into the pre-trained CLIP model V, respectively. qua Image encoders and text encoders, as well as the semantic CLIP model V sem The pre-trained CLIP model V is quantified by calculating the KL divergence between zero-sample semantic prediction probabilities. qua The semantic attention level in the model is obtained by calculating the cross-entropy between the model's predictions and the basic facts to obtain the quality optimization direction of the IQA task. Finally, the gradient of the IQA task is adjusted by pruning gradients aligned with the general semantic direction in the shared representation space to generate regularized gradients for backward updates, thus balancing the learning between quality and semantics.

Images