Blind image quality assessment algorithm based on multi-scale relation perception adaptor

Abstract

The application discloses a blind image quality evaluation algorithm based on a multi-scale relationship perception adapter, and the core is parameter efficient fine-tuning and spatial block relationship modeling. The steps comprise the following steps. First, input a distorted image and a text prompt into a pre-trained backbone network with frozen parameters, and extract multi-level spatial basic features. Then, aggregate multi-scale features in a lightweight adapter, calculate a similarity matrix between blocks through a relationship transformation module, and realize feature reconstruction of the interaction between local distortion and global quality perception. Finally, perform cross-modal fusion guided by a text feature, and output an objective quality score through a regression network. The algorithm has low fine-tuning cost, complete pre-training knowledge reservation, and high evaluation accuracy for complex non-uniform distortion.

Description

Technical Field

[0001] This invention relates to a deep learning-based image quality assessment algorithm, which is based on cutting-edge technologies in the field of image processing, and particularly focuses on no-reference (blind) image quality assessment based on efficient fine-tuning of large model parameters (PEFT), belonging to the fields of computer vision and image processing. Background Technology

[0002] Blind Image Quality Assessment (BIQA) is a technique designed to enable computers to simulate the human visual system (HVS) and automatically and objectively evaluate the quality of distorted images in the absence of a distortion-free reference image. In practical applications (such as video streaming, image enhancement, and social media sharing), complex, mixed, and often unevenly distributed distortions (such as local blurring, blockiness, and noise) are introduced during image acquisition, compression, and transmission, significantly impacting image usability and visual experience. Therefore, accurate BIQA technology has significant application value.

[0003] With the development of deep learning technology, especially large-scale vision-language models (such as CLIP), blind image quality assessment models have demonstrated feature extraction and generalization capabilities that surpass traditional convolutional neural networks. These models typically utilize powerful Vision Transformer (ViT) backbone networks to extract image features and combine them with text prompts for quality prediction. However, existing unified assessment algorithms mainly face the following technical bottlenecks: insufficient modeling of the relationship between local distortion and global perception: While traditional ViT backbones interact with global features through self-attention mechanisms, their focus is primarily on high-dimensional semantic features, lacking specific spatial relationship transformations for "non-uniform physical distortion." In the human visual system, severe distortion in a specific local area (such as blurring of the face region) often has a decisive impact on the overall image quality score. Existing models lack explicit modeling of this "cross-influence of local patch distortion on global quality." Fine-tuning is costly and can easily destroy pre-trained knowledge: If a new relation modeling module is forcibly added directly into the ViT backbone network and full fine-tuning is performed, not only will it consume a lot of computational resources, but it will also easily cause catastrophic forgetting and destroy the rich visual-language alignment knowledge learned by the large model in the pre-training stage. Summary of the Invention

[0004] The purpose of this invention is to disclose a blind image quality assessment algorithm based on a multi-scale relational perception adapter. The algorithm aims to improve the accuracy and generalization ability of the model for assessing the quality of complex and non-uniformly distorted images by explicit feature space transformation and tile relation modeling without destroying the pre-trained weights of the large model, and to expand its application in various practical image processing pipelines.

[0005] In embodiments according to this disclosure, the blind image quality assessment algorithm based on a multi-scale relation-aware adapter includes the following steps:

[0006] Step 1: Use the preprocessing module to obtain the distorted image to be evaluated and divide it into patches. Input the text prompts for quality evaluation into the visual-language pre-trained backbone network with frozen parameters to extract multi-level spatial basic features.

[0007] Step 2: Input the extracted multi-level spatial basic features into the relation-aware adapter. Through the multi-scale feature aggregation module, project and align the features at different depths according to the channel dimension to generate a fused feature sequence of the same scale.

[0008] Step 3: Construct a spatial relationship transformation module inside the relationship-aware adapter, use linear transformation to calculate the similarity relationship matrix between each feature patch, dynamically reconstruct the features of each patch through the relationship matrix, and explicitly model the mutual influence of local distortion on global perception.

[0009] Step 4: Using text prompts as guidance queries, perform cross-modal cross-attention calculation on the reconstructed relation features, input the fused features into the quality regression network, and output the final image quality prediction score.

[0010] Further technical solutions, step 2 specifically includes the following steps:

[0011] Step 2.1: Extract the output feature maps F from the i-th layer, j-th layer, and the last L-th layer of the backbone network, respectively. i F j F L To capture different levels of information, from low-level texture distortion to high-level semantic distortion.

[0012] Step 2.2: Align the number of channels in the extracted feature map using convolutional layers and normalization operations.

[0013] Step 2.3: Concatenate the aligned multi-scale features along the sequence dimension, and obtain the fused feature sequence X through a linear mapping layer. agg .

[0014] Further technical solutions, step 3 specifically includes the following steps:

[0015] Step 3.1: Aggregate the multi-scale feature sequence X agg The query matrix Q, key matrix K, and value matrix V are generated through three independent linear mapping layers.

[0016] Step 3.2: To capture the distortion impact weights between different tiles in space, calculate the inner product of Q and the transpose of K, and divide by the scaling factor. (Where d is the feature dimension), and finally, it is normalized using the Softmax activation function to generate the global spatial relation matrix R. The calculation formula is as follows:

[0017]

[0018] Step 3.3: Perform matrix multiplication between the relation matrix R and the value matrix V to obtain the feature sequence F after global relation-aware adjustment. rel The calculation formula is as follows:

[0019] F rel =R·V

[0020] Further technical solutions, step 4 specifically includes the following steps:

[0021] Step 4.1: Encode the text prompt features F using the language model. prompt As a query, the relational feature F obtained in step 3 is used. rel As keys and values, they are fed into the cross-attention network layer for cross-modal fusion.

[0022] Step 4.2: Perform global pooling or flattening on the output features after cross-modal fusion, and input them into a quality regression network consisting of multiple fully connected layers (MLP).

[0023] Step 4.3: Map the image to a specific evaluation range using a non-linear activation function to output the final objective image quality score.

[0024] The present invention provides a blind image quality assessment algorithm based on a multi-scale relation-aware adapter, which has the following beneficial effects:

[0025] 1) This invention achieves efficient parameter fine-tuning and pre-training knowledge preservation. The framework innovatively adopts a "frozen backbone network + lightweight adapter" architecture. All complex feature relationship reconstructions are completed in the external adapter, while the parameters of the backbone network remain unchanged. This design significantly reduces the computational cost and time required for model training, while perfectly preserving the powerful multimodal representation capabilities obtained by large models pre-training on massive amounts of image and text data.

[0026] 2) This invention achieves accurate modeling of the relationship between local distortion and global perception. By introducing an explicit patch relation matrix into the adapter, the similarity and influence weights between features of different regions of the image are calculated. The model can intelligently perceive human visual characteristics such as "severe local distortion has a fatal impact on overall quality", significantly improving the evaluation accuracy of non-uniform distortion.

[0027] 3) This invention employs a multi-scale feature fusion and prompting guidance strategy. By aggregating features from the shallow, middle, and deep layers of the backbone network, the model can simultaneously capture texture noise information from the lower layers and structural semantic information from the higher layers. Furthermore, it utilizes textual instructions to dynamically adjust the feature fusion weights, giving the algorithm strong task adaptability and generalization applicability. Attached Figure Description

[0028] Figure 1 The flowchart is a process for a blind image quality assessment algorithm based on a multi-scale relational sensing adapter, which is the subject of this invention.

[0029] Figure 2 This is a schematic diagram of the network structure of the blind image quality assessment algorithm based on a multi-scale relational perception adapter, which is involved in this invention. Detailed Implementation

[0030] This invention proposes a blind image quality assessment algorithm based on a multi-scale relation-aware adapter, utilizing efficient parameter fine-tuning techniques and spatial relation modeling to achieve accurate evaluation of distorted images. To more clearly illustrate the purpose, technical solution, and advantages of this invention, the technical solution will be described in detail below with reference to the accompanying drawings. It should be noted that the specific embodiments described below are only for explaining the principles of this invention and are not intended to limit the scope of implementation of this invention.

[0031] This invention first provides a blind image quality assessment algorithm, the specific flowchart of which is shown below. Figure 1 This includes the following steps:

[0032] Step 1: Use the preprocessing module to obtain the distorted image to be evaluated and divide it into patches. Input the text prompts for quality evaluation into the visual-language pre-trained backbone network with frozen parameters to extract multi-level spatial basic features.

[0033] Step 2: Input the extracted multi-level spatial basic features into the relation-aware adapter. Through the multi-scale feature aggregation module, project and align the features at different depths according to the channel dimension to generate a fused feature sequence of the same scale.

[0034] Step 3: Construct a spatial relationship transformation module inside the relationship-aware adapter, use linear transformation to calculate the similarity relationship matrix between each feature patch, dynamically reconstruct the features of each patch through the relationship matrix, and explicitly model the mutual influence of local distortion on global perception.

[0035] Step 4: Using text prompts as guidance queries, perform cross-modal cross-attention calculation on the reconstructed relation features, input the fused features into the quality regression network, and output the final image quality prediction score to obtain the enhanced normal illumination image.

[0036] In a preferred embodiment of the present invention, the "visual-language pre-trained backbone network" described in step 1 is mainly used for basic feature extraction of the input distorted image. It should be noted that the present invention does not strictly limit the specific network structure of this backbone network. In specific implementations, the backbone network can be a visual model based on the Transformer architecture (such as ViT, Swin Transformer, etc.) or a model based on a convolutional neural network (such as ResNet, etc.). Preferably, this embodiment uses a CLIP model visual encoder (such as CLIP-ViT) as the backbone network. After the input image is divided into several local patches, it is processed by a linear embedding layer with positional encoding. Throughout the training and fine-tuning phases, the weight parameters of this backbone network are set to a frozen state and do not participate in gradient updates.

[0037] In a preferred embodiment of the present invention, step 2 constructs a multi-scale feature aggregation module. The model extracts features from different depths of the backbone network (e.g., from layers 4, 8, and 12 of a 12-layer ViT). One-dimensional convolutional (Convld) or linear projection layers are used to normalize and align the number of feature channels at different levels. These features are then concatenated along the sequence length dimension to form a comprehensive feature sequence X containing local texture defects and global semantic degradation. agg .

[0038] In a preferred embodiment of the present invention, a spatial relationship transformation module is specifically constructed in step 3. Given an input feature sequence X... agg The forward propagation of this serialization layer structure can be represented as:

[0039] First, calculate the query matrix Q, the key matrix K, and the value matrix V:

[0040] Q = X agg W q

[0041] Q = X agg W k

[0042] Q = X agg W v

[0043] Among them W q W k W v Let R be a learnable linear projection weight matrix. Then, calculate the spatial relation matrix R:

[0044]

[0045] Finally, the output feature F is reconstructed through relation reconstruction. rel :

[0046] F rel =R·V

[0047] This step uses explicit matrix operations to integrate the features of each image patch with the distortion information of other global patches, thus completing the reconstruction of the relationship from local to global.

[0048] In a preferred embodiment of the present invention, step 4 utilizes a pre-set text prompt (e.g., "Evaluate the quality of this image") and the feature F extracted by the text encoder. prompt Construct a cross-attention mechanism to integrate F... prompt Mapping to a Query will reconstruct feature F rel The data is mapped to Key and Value. The fused multimodal features are then input into a multilayer perceptron (MLP) consisting of fully connected layers, ultimately regressing a scalar quality score.

[0049] The above description mentions various methods of combining technical features, but does not list all possible combinations. However, it should be emphasized that as long as the combination of these technical features is reasonable in implementation and does not contradict each other, it should be considered within the scope of this specification.

Claims

1. A blind image quality assessment algorithm based on a multi-scale relation-aware adapter, characterized in that, The method includes the following steps: Step 1: Obtain the distorted image to be evaluated, perform preprocessing and patch division, and input the text prompts for quality evaluation into the visual-language pre-trained backbone network with frozen parameters to extract multi-level spatial basic features. Step 2: Input the extracted multi-level spatial basic features into the relation-aware adapter. First, through the multi-scale feature aggregation module, project and align the features at different depths according to the channel dimension to generate a fused feature sequence of the same scale. Step 3: Construct a spatial relationship transformation module inside the relationship-aware adapter, use linear transformation to calculate the similarity relationship matrix between each feature patch, dynamically reconstruct the features of each patch through the relationship matrix, and explicitly model the mutual influence of local distortion on global perception. Step 4: Using text prompts as guidance queries, perform cross-modal cross-attention calculation on the reconstructed relation features, input the fused features into the quality regression network, and output the final image quality prediction score.

2. The algorithm according to claim 1, characterized in that, Step 1 specifically includes: cutting the input distorted image into a non-overlapping sequence of local patches, adding position encoding after passing through a linear embedding layer, and inputting it into a Transformer backbone network containing a multi-layer multi-head self-attention mechanism (MSA) and a multi-layer perceptron (MLP); during training and inference, the weight parameters of the backbone network remain frozen and are not updated.

3. The algorithm according to claim 1, characterized in that, The specific calculation process of the multi-scale feature aggregation module in step 2 is as follows: Output features F are extracted from the i-th layer, j-th layer, and the last L-th layer of the backbone network. i F j F L The number of channels in the feature maps is aligned using convolutional layers and normalization operations. Then, through concatenation and linear mapping, a multi-scale aggregated feature sequence X, which integrates low-level texture distortion and high-level semantic distortion, is obtained. agg .

4. The algorithm according to claim 1, characterized in that, The spatial relationship transformation module in step 3 specifically includes the following sub-steps: Step 3.1 Linear Projection: Project the multi-scale aggregated feature sequence X agg The query matrix Q, key matrix K, and value matrix V are generated through three independent linear mapping layers. Step 3.2 Relationship Matrix Calculation: To capture the distortion impact weight of patch m on patch n, calculate the inner product of the transposes of Q and K, and divide by the scaling factor. Finally, the global spatial relation matrix R is generated by normalization using an activation function. The calculation formula is as follows: Step 3.3 Relation Reconstruction and Feature Output: Perform matrix multiplication on the relation matrix R and the value matrix V to obtain the feature sequence F after global relation-aware adjustment. rel The calculation formula is as follows: F rel =R·V This module redistributes and enhances previously isolated local distortion features within a global perspective.

5. The algorithm according to claim 1, characterized in that, Step 4 specifically involves: encoding the text prompt features F using a language model. prompt As a query, the relational feature F obtained in step 3 will be used. rel As the key and value, they are fed into the cross-attention network layer for cross-modal fusion. Finally, the output features are flattened and input into the quality regression network composed of multiple fully connected layers. They are mapped to a specific evaluation range through a non-linear activation function, and the final image objective quality score is output.

Images