A method and device for image quality assessment without reference based on visual interactive perception dual-stream network
The semantic and distortion features of the image are extracted and fused through the Visual Interaction Perception Two-Stream Network (VIPNet), which solves the problem that existing models fail to consider the interaction between image content and distortion, and achieves more accurate reference-free image quality assessment.
Patent Information
- Application Number
- CN202211189584.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-09-28
- Publication Date
- 2025-09-05
- Estimated Expiration
- 2042-09-28
AI Technical Summary
Existing no-reference image quality assessment models fail to effectively consider the visual interaction between image content and distortion, resulting in insufficient accuracy in image quality assessment.
A method based on the visual interaction perception two-stream network (VIPNet) is adopted. The semantic features and distortion perception features of the image are extracted respectively through the pre-trained content perception module, distortion perception module and visual interaction module. The features are then fused through the visual interaction module, and finally the image quality is evaluated based on the fused features.
The accuracy of image quality assessment is improved, which can better reflect the interactive impact between image content and distortion, and enhance the performance of no-reference image quality assessment.
Smart Images
Figure CN115457015B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a method and device for image quality assessment without reference based on a visual interactive perception dual-stream network, belonging to the technical field of image quality assessment. Background Art
[0002] A large number of images are uploaded and downloaded every day on social platforms and visual media applications. However, the perceived quality of images can be compromised at various stages of image processing, including acquisition, compression, transmission, and storage. Therefore, it is crucial to stably control image quality in processing systems. Image quality assessment (IQA) algorithms are generally categorized into three types based on the amount of available reference information: full-reference, semi-reference, and no-reference (blind IQA, BIQA). Since reference information is unavailable in many application scenarios, no-reference image quality assessment, which operates directly on images, has been studied more intensively in recent years due to its wider demand.
[0003] The goal of objective image quality assessment is to automatically evaluate the perceptual quality of an image in a manner consistent with human subjective judgment. To construct an IQA dataset, many observers are invited to evaluate the visual quality of distorted images. The perceptual quality of each image is then annotated using the mean opinion score (MOS or DMOS). Traditional BIQA models are typically designed based on carefully designed handcrafted features that are mapped to MOS values via a nonlinear regression model. However, due to the complexity of distortion and image content, the quality perception capabilities of handcrafted features remain limited.
[0004] In recent years, due to the powerful representational capabilities of deep neural networks (DNNs), an increasing number of deep learning-based BIQA models have been proposed. These mainstream models fall into two categories: those based on distortion intensity ranking and those based on deep semantic features. During the training process of distortion intensity ranking models, a library of distorted images is synthesized using a range of distortion intensities. The quality of the distorted images is then ranked according to their corresponding distortion intensity for model training. However, for the same original image, the quality ranking of distorted images is highly correlated with the distortion intensity, while for images with different content, the correlation between image quality ranking and distortion intensity is generally weaker. Models based on deep semantic features often utilize pre-trained weights from image classification datasets (such as ImageNet) as initialization for the feature extraction backbone network. These efforts attempt to map the degree of semantic feature degradation to image quality. However, semantic feature degradation does not necessarily reflect a decrease in image quality. This is because image classification tasks require robustness to noise. In particular, noise is often added during classification training for data augmentation. Under these circumstances, models based on deep semantic features are often insensitive to some distortions that still result in image quality degradation.
[0005] The principle of visual masking effect states that when observing images, the human visual system (HVS) has different distortion visibility for images with the same distortion but different content. The HVS's perception of image distortion is affected by the interaction between image content and distortion. However, neither distortion strength ranking-based models nor deep semantic feature-based models take this interaction into account. Specifically, distortion strength ranking-based models mainly use distortion strength to rank image quality, while semantic-based methods only focus on the degree of damage to deep semantic features as a representation of image content quality. Summary of the Invention
[0006] The purpose of the present invention is to propose a reference-free image quality assessment method and device based on a visual interaction perception two-stream network (VIPNet), which evaluates image quality from the perspective of image distortion visibility, which is modeled as the visual interaction between image content and corresponding distortion.
[0007] In order to achieve the above object, the technical solution adopted by the present invention is as follows:
[0008] The present invention provides a method for image quality assessment without reference based on a visual interactive perception dual-stream network, comprising:
[0009] Preprocess the image;
[0010] A pre-trained content-aware module is used to extract semantic features from pre-processed images;
[0011] Use the pre-trained distortion perception module to extract distortion perception features from the pre-processed image;
[0012] The semantic features and distortion perception features of the extracted images are fused using a pre-trained visual interaction perception module to obtain fused features.
[0013] Image quality assessment is performed based on the fusion features to obtain the image quality score.
[0014] Furthermore, the image preprocessing includes:
[0015] Crop the image to a uniform size.
[0016] Furthermore, the pre-trained content-aware module includes:
[0017] Imagenet is used as the training set, and the image classification network ResNet50 is input without the last global average pooling layer and the fully connected layer, and the output feature map of the tth module of ResNet50 is obtained. t∈1,2,3,4;
[0018] The feature map obtained Each input has a 1×1 kernel, a 1×1 stride, and an output channel size of c t Convolutional layers;
[0019] The results of each convolutional layer are downsampled to the same size using an adaptive pooling layer;
[0020] Connect the output features of the adaptive pooling layer as multi-scale image semantic features;
[0021] Imagenet is used to train the network weights of the image classification network ResNet50. After the training is completed, the ResNet50 is removed from the modules with the last global average pooling layer and the fully connected layer, as well as the network with a 1×1 kernel, a 1×1 step size, and an output channel size of c. t The structure composed of convolutional layers and adaptive pooling layers is used as the content-aware module.
[0022] Furthermore, the output channel size c t The values are: c1=128, c2=256, c3=512, c4=1152.
[0023] Furthermore, the pre-trained distortion perception module includes:
[0024] Construct a distorted image training sample set;
[0025] Convert each distorted image training sample into YCbCr color space as input for BoTNet training;
[0026] After training is completed, the last global average pooling layer and linear layer are removed to obtain the distortion perception module.
[0027] Furthermore, the constructing of the distorted image training sample set includes:
[0028] Select N original images from the KADIS-700K database;
[0029] Generate distorted image samples with 30 distortion types and 5 distortion intensities for each original image;
[0030] A distorted image training sample set is composed of 150×N distorted image samples; the 30 distortion types include: pink noise, lossy compression, contrast change, underexposure, overexposure, and the same 25 distortion types as the KADID-10K database.
[0031] Furthermore, the pre-trained visual interaction perception module includes:
[0032] The trained content-aware module is used to extract the semantic features of the training samples, and the trained distortion-aware module is used to extract the distortion-aware features of the training samples. The semantic features and distortion-aware features of the training samples are connected to obtain F, where c d represents the output channel size of the distortion-aware feature, w and h represent the width and height of the output feature respectively;
[0033] Flatten F and map it to the feature sequence through convolution operation in l = w × h is the length of the feature sequence, and d is the size of the feature sequence;
[0034] A learnable marker Add to And add the position embedding PE to get the characteristics of the training sample
[0035] The features of the training samples are sequentially input into the feature fusion module and the multi-layer perceptron block of the multi-head self-attention mechanism for training, and finally a trained visual interaction perception module is obtained.
[0036] Furthermore, the image quality assessment based on the fusion features to obtain the image quality score includes:
[0037] The final fusion feature is passed through a fully connected layer to output the quality prediction score.
[0038] The present invention also provides an image quality assessment device without reference based on a visual interactive perception dual-stream network, comprising:
[0039] Content-aware module, used to extract semantic features of images;
[0040] Distortion perception module, used to extract distortion perception features of images;
[0041] Visual interaction module, used to fuse the semantic features and distortion perception features of the image;
[0042] An evaluation module is used to evaluate the image quality based on the fusion features and obtain an image quality score;
[0043] The content perception module is pre-trained using the aforementioned image no-reference quality assessment method based on visual interaction perception two-stream network;
[0044] The distortion perception module is pre-trained using the aforementioned image no-reference quality assessment method based on visual interactive perception two-stream network;
[0045] The visual interaction module is pre-trained using the aforementioned image no-reference quality assessment method based on visual interaction perception two-stream network.
[0046] Furthermore, the evaluation module adopts a fully connected layer.
[0047] The beneficial effects of the present invention are:
[0048] The proposed no-reference image quality assessment method based on a visually interactive perception dual-stream network models the interaction between image content and image distortion based on the principle of visual masking, effectively improving image quality assessment performance. During module training, the present invention employs a two-stage training approach, enabling the model to explicitly extract feature representations of both image content and image distortion. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] Figure 1 The present invention provides an architecture for a device for image quality assessment without reference based on a dual-stream network of visual interactive perception. DETAILED DESCRIPTION
[0050] The present invention will be further described below. The following examples are only used to more clearly illustrate the technical solution of the present invention and are not intended to limit the scope of protection of the present invention.
[0051] Example 1
[0052] This embodiment provides a non-reference image quality assessment device based on a visual interactive perception dual-stream network. Figure 1 , including content perception module (CPM), distortion perception module (DPM) and visual interaction module (VIM).
[0053] The content-aware module is used to extract the semantic features of the image as a perception of the image content.
[0054] The distortion perception module is used to extract the distortion perception features of the image.
[0055] The visual interaction module is used to fuse the semantic features of the image and the distortion perception features of the image;
[0056] The evaluation module is used to evaluate the image quality based on the fusion features and obtain an image quality score.
[0057] In this embodiment, the content perception module is generated by training an image classification network (such as ResNet50) without the last global average pooling layer and the fully connected layer.
[0058] Those skilled in the art should know that ResNet50 mainly includes four modules, each of which is composed of stacking different numbers of residual blocks, and finally classifies the image through a global average pooling layer and a fully connected layer.
[0059] In this embodiment, Imagenet is used to train ResNet50. The training process is as follows:
[0060] Input the image of the training set into ResNet50 to obtain the output feature map of the tth module where t∈1,2,3,4;
[0061] The feature map obtained Feed it with a 1×1 kernel, 1×1 stride and output channel size c t Convolutional layers;
[0062] The convolution results of each module are downsampled to the same size using an adaptive pooling layer;
[0063] The output features of the adaptive pooling layer of each module are connected as multi-scale image semantic features, which are expressed as:
[0064]
[0065] in, c s =Σ t c t , w, and h are the channel size, width, and height of the output image semantic features, respectively. The operations Concat{·}, Pooling(·), and Conv(·) represent concatenation, adaptive pooling, and convolution, respectively.
[0066] In this embodiment, the values of the output channels are 128, 256, 512, and 1152 respectively.
[0067] It should be noted that the size of the output feature map is 1 / 32 of the input image size, where 32 is determined by the ResNet50 structure.
[0068] Train the ResNet50 network weights. After the training is completed,
[0069] The ResNet50 network removes the last global average pooling layer and the four fully connected layers, and the above-mentioned modules with 1×1 kernel, 1×1 stride and output channel size c t The structure composed of convolutional layers and adaptive pooling layers is used as the content-aware module.
[0070] In this embodiment, the distortion perception module is generated by BoTNet training with the last global average pooling layer and linear layer removed.
[0071] In this embodiment, the distortion perception module training process is as follows:
[0072] First, a large-scale synthetic distortion dataset was generated. Specifically, 40,000 original images were selected from the KADIS-700K database. For each original image, distorted image samples were generated with 30 distortion types and 5 distortion intensities for each distortion type. Twenty-five of the distortion types are the same as those in the KADIS-10K database, and the other five types include pink noise, lossy compression, contrast change, underexposure, and overexposure. Therefore, each original image has 150 distorted image samples (i.e., 30 distortion types x 5 distortion intensities), for a total of 40,000 × 150 distorted image samples constituting the training set.
[0073] Those skilled in the art should know that KADIS-700K and KADID-10K are both existing synthetic distortion IQA databases.
[0074] BoTNet, a hybrid design based on spatial convolution and multi-head self-attention (MHSA), is used as the backbone network for distortion classification task training. BoTNet replaces the convolutional layers in the last three bottleneck blocks of ResNet with MHSA.
[0075] Convert each distorted image sample to YCbCr color space as input, expressed as:
[0076]
[0077] Where Y represents brightness information, C b and C r Represents chromaticity information, R, G, and B are the color information of the distorted image samples respectively.
[0078] For the input image x and its corresponding classification label y, where y∈{c|0≤c≤C-1,c∈N +}, c is the distortion category of the distorted image, and C is the number of distortion categories in the generated distorted image dataset. In this embodiment, C is 150.
[0079] The output of the BoTNet-based distortion classification model is expressed as:
[0080]
[0081] in represents the predicted probability of the cth class,
[0082] The definition is as follows:
[0083]
[0084] where φ θ represents a distorted classification network with network parameters θ, [·] c Refers to the cth element in the probability vector. Softmax() means normalizing the output vector to between 0 and 1.
[0085] In order to train the distortion classification model, this embodiment uses the cross entropy loss function Minimize, that is:
[0086]
[0087] Where 1[y=c] represents the one-hot encoding of the label.
[0088] The generated large-scale synthetic distortion dataset is used to train BoTNet, and the BoTNet with the last average pooling layer and linear layer removed is used as the distortion perception module (DPM).
[0089] For the input image, the final feature spectrum of the distortion perception module DPM is extracted as the distortion perception representation, denoted as F DPM ,in c d , w, and h represent the channel size, width, and height of the output features, respectively.
[0090] In this embodiment, the visual interaction perception module is used to perform feature fusion on the semantic features and distortion perception features of the image based on the self-attention mechanism.
[0091] The visual interaction perception module is specifically used to:
[0092] The semantic features of the image F CPM and distortion perception feature F DPM The features of are connected to get F, where
[0093] Flatten F and map it to the feature sequence through convolution operation in l = w × h and d are the length and size of the sequence respectively.
[0094] A learnable marker Add to And add position embedding PE (Position Embedding), so that the model can better perceive the contribution of different areas of the image to quality prediction, where
[0095] In the self-attention based feature fusion block (SAFFB), three learnable weight matrices are defined to Converted into three different sequence vectors (query Q, keyword K and value V), expressed as:
[0096]
[0097] in And d q =d k =d v =d′ is the dimension of the corresponding sequence vector.
[0098] Through Softmax Calculate the weight of feature interaction, expressed as:
[0099]
[0100] In order to extract various complex relationships between different elements in the input sequence, multi-head self-attention consists of multiple self-attention blocks, which will add tags The features after position embedding PE are linearly projected into multiple feature subspaces and processed in parallel by independent attention blocks. The result vectors of multiple SAFFBs are concatenated and mapped to the final output, which is expressed as:
[0101]
[0102] where H is the number of SAFFBs, is the output linear projection matrix.
[0103] The present invention adds a multi-layer perceptron (MLP) block after Multi-SAFFB, uses Layer Normalization (LN) to perform layer normalization before each block, and adds residual connections between them.
[0104] The VIM process can be simply described as:
[0105]
[0106] Where m = 1, 2, ... M, M is the depth of the model, when m = 1, z m-1 =z0.
[0107] The MLP contains two layers of GELU activation functions.
[0108] In this embodiment, the evaluation module is specifically used to evaluate the fusion feature z m A fully connected layer is used to output the quality prediction score.
[0109] This embodiment provides semantic features and distortion features based on CPM and DPM, and the goal of VIM is to model the interaction between the two. However, the subjective quality score cannot explicitly guide the model to extract feature representations of content and distortion. Updating the parameters of modules CPM and DPM while training VIM will damage the image content and distortion features. Therefore, in order to better establish an interactive model, this embodiment adopts a stage-based training method: in the first stage, we generate a large-scale synthetic distortion database and obtain a distortion perception module by constructing a distortion classification task; secondly, the pre-trained image classification network Resnet50 is used as the content perception module; in the second stage, the visual interaction module models the visual interaction relationship between image content and distortion through a feature fusion module based on the self-attention mechanism. During the model optimization process, the pre-trained weights of the content perception module and the distortion perception module are frozen, and the parameters of the visual interaction model are only updated through backpropagation.
[0110] It should be noted that this embodiment is implemented on the pytorch experimental platform.
[0111] Example 2
[0112] This embodiment provides a method for image quality assessment without reference based on a visual interactive perception dual-stream network, including:
[0113] S1, preprocess the image;
[0114] S2, extracts semantic features of the image using a pre-trained content-aware module;
[0115] S3, extracting the distortion perception features of the image using a pre-trained distortion perception module;
[0116] S4, using the pre-trained visual interaction module to fuse the semantic features and distortion perception features of the extracted image to obtain fused features;
[0117] S5. Perform image quality assessment based on the fused features to obtain an image quality score.
[0118] In this embodiment, preprocessing the image includes:
[0119] Crop the image to a uniform size, such as 288×384.
[0120] In this embodiment, the content perception module is trained based on the method in Example 1.
[0121] In this embodiment, the distortion perception module is trained based on the method in Example 1.
[0122] In this embodiment, the visual interaction module is trained based on the method in Example 1.
[0123] In this embodiment, a quality prediction score is output through a fully connected layer.
[0124] It is worth noting that the device embodiment corresponds to the above-mentioned method embodiment, and the implementation methods of the above-mentioned method embodiments are applicable to the device embodiment and can achieve the same or similar technical effects, so they will not be repeated here.
[0125] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.
[0126] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0127] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.
[0128] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.
[0129] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the scope of protection of the claims of the present invention.
Claims
1. A no-reference image quality assessment method based on a visual interactive perception two-stream network, characterized by: include: Preprocess the image; A pre-trained content-aware module is used to extract semantic features from pre-processed images; Use the pre-trained distortion perception module to extract distortion perception features from the pre-processed image; The semantic features and distortion perception features of the extracted images are fused using a pre-trained visual interaction perception module to obtain fused features. Perform image quality assessment based on fusion features to obtain image quality scores; Among them, the pre-trained visual interaction perception module includes: The trained content-aware module is used to extract the semantic features of the training samples, and the trained distortion-aware module is used to extract the distortion-aware features of the training samples. The semantic features and distortion-aware features of the training samples are connected to obtain F, where c s represents the output channel size of semantic features, c d represents the output channel size of the distortion-aware feature, w and h represent the width and height of the output feature respectively; Flatten F and map it to the feature sequence through convolution operation in l = w × h is the length of the feature sequence, and d is the size of the feature sequence; A learnable marker Add to And add the position embedding PE to get the characteristics of the training sample The features of the training samples are sequentially input into the feature fusion module and the multi-layer perceptron block of the multi-head self-attention mechanism for training, and finally a trained visual interaction perception module is obtained.
2. The image quality assessment method without reference based on a visual interactive perception dual-stream network according to claim 1 is characterized in that: The image preprocessing comprises: Crop the image to a uniform size.
3. The image quality assessment method without reference based on a visual interactive perception dual-stream network according to claim 1 is characterized in that: Pre-trained content-aware modules include: Imagenet is used as the training set, and the image classification network ResNet50 is input without the last global average pooling layer and the fully connected layer, and the output feature map of the tth module of ResNet50 is obtained. t∈1,2,3,4; The feature map obtained Each input has a 1×1 kernel, a 1×1 stride, and an output channel size of c t Convolutional layers; The results of each convolutional layer are downsampled to the same size using an adaptive pooling layer; Connect the output features of the adaptive pooling layer as multi-scale image semantic features; Imagenet is used to train the network weights of the image classification network ResNet50. After the training is completed, the ResNet50 is removed from the modules with the last global average pooling layer and the fully connected layer, as well as the network with a 1×1 kernel, a 1×1 step size, and an output channel size of c. t The structure composed of convolutional layers and adaptive pooling layers is used as the content-aware module.
4. The image quality assessment method without reference based on a visual interactive perception dual-stream network according to claim 3 is characterized in that: The output channel size c t The values are: c1=128, c2=256, c3=512, c4=1152.
5. The image quality assessment method without reference based on a visual interactive perception dual-stream network according to claim 1 is characterized in that: The pre-trained distortion-aware module includes: Construct a distorted image training sample set; Convert each distorted image training sample into YCbCr color space as input for BoTNet training; After training is completed, the last global average pooling layer and linear layer are removed to obtain the distortion perception module.
6. The image quality assessment method without reference based on a visual interactive perception dual-stream network according to claim 5 is characterized in that: The step of constructing a distorted image training sample set includes: Select N original images from the KADIS-700K database; Generate distorted image samples with 30 distortion types and 5 distortion intensities for each original image; A distorted image training sample set is composed of 150×N distorted image samples; the 30 distortion types include: pink noise, lossy compression, contrast change, underexposure, overexposure, and the same 25 distortion types as the KADID-10K database.
7. The image quality assessment method without reference based on a visual interactive perception dual-stream network according to claim 1 is characterized in that: The image quality assessment based on the fusion features to obtain the image quality score includes: The final fusion feature is passed through a fully connected layer to output the quality prediction score.
8. A device for image quality assessment without reference based on a visual interactive perception dual-stream network, characterized in that: include: Content-aware module, used to extract semantic features of images; Distortion perception module, used to extract distortion perception features of images; Visual interaction module, used to fuse the semantic features and distortion perception features of the image; An evaluation module is used to evaluate the image quality based on the fusion features and obtain an image quality score; The content perception module is pre-trained using the image no-reference quality assessment method based on the visual interactive perception dual-stream network according to any one of claims 1 to 4; The distortion perception module is pre-trained using the image no-reference quality assessment method based on visual interactive perception dual-stream network according to claim 5 or 6; The visual interaction module is pre-trained using the image no-reference quality assessment method based on visual interaction perception dual-stream network described in claim 1.
9. The image quality assessment device without reference based on a visual interactive perception dual-stream network according to claim 8, characterized in that: The evaluation module adopts a fully connected layer.