Concentrated attention-based no-reference image quality evaluation method

Abstract

The invention discloses a no-reference image quality evaluation method based on concentrated attention, and the method comprises the steps: constructing a no-reference image quality evaluation system based on concentrated attention at an offline stage, extracting the features of an input image, obtaining a weight matrix corresponding to each window through window division and representative token extraction processing, and obtaining a weight matrix corresponding to each window; and the local information and the global information are fused after being superposed with the features coded by the local spatial information, and are used for carrying out quality evaluation on the image in an online stage after being trained. According to the method, the advantages of convolution and an attention mechanism are combined, the mode of generating representatives through window division is adopted, long-distance dependence between elements is captured with extremely small calculation amount, and global information can be added to each element in a window according to needs through an attention module in the window.

Description

Technical Field

[0001] The present invention relates to a technology in the field of image processing, specifically a reference-free image quality assessment method based on Espresso Attention (IQA). Background Art

[0002] Existing networks such as Vision Transformer for image quality assessment can significantly improve the image evaluation performance of traditional convolutional models when facing complex real distortions. However, due to the computational complexity of multi-head attention, Transformer networks often require more computing resources and inference time than CNN. In order to better combine the advantages of CNN and Transformer, many hybrid structures have been proposed for image recognition, object detection and semantic segmentation tasks. However, the performance improvement of these hybrid structure models in image quality assessment is still very limited. Summary of the invention

[0003] In view of the problem that the existing technology cannot be generalized on the basis of various backbone networks and requires image preprocessing, which leads to additional complexity, the present invention proposes a reference-free image quality assessment method based on concentrated attention. By combining the advantages of convolution and attention mechanisms, the method of generating representatives by dividing windows is used to capture the long-distance dependencies between elements with very little computational effort. Through the in-window attention module, each element in the window can add global information as needed. The method can be applied to different types of the latest CNN / Transformer network backbones and significantly improve the performance of the network backbone in image quality assessment.

[0004] The present invention is achieved through the following technical solutions:

[0005] The present invention relates to a reference-free image quality assessment method based on concentrated attention. After extracting the features of an input image and encoding local information to obtain a coded feature map of local information interaction, the coded feature map is divided into windows and representative tokens are generated in each window. The representative tokens are used to replace the original complete feature map to perform multi-head self-attention calculation and obtain an updated feature map. The quality assessment result is obtained by merging the updated feature map with the coded feature map of local information interaction after allocating attention within the window through regression calculation.

[0006] The present invention relates to a reference-free image quality assessment system based on concentrated attention for implementing the above method, comprising: a feature extraction module, an adjustment module, a local space information encoding module, a representative token generation module, a global attention calculation module, a replication module, an in-window attention module and a quality assessment module, wherein: the feature extraction module extracts features from an input image; the adjustment module adjusts the channel dimension of a feature map; the local space information encoding module encodes local information on the feature map; the representative token generation module divides the feature map into windows and generates representative tokens for each window; the global attention calculation module calculates global self-attention in a low-cost manner based on the representative token; the replication module replicates the token according to the window size represented by the token; the in-window attention module allocates in-window attention weights to the feature map obtained after replication; and the quality assessment module performs regression calculation based on the result of adding the feature map after attention allocation and the feature map after local information interaction to obtain a quality assessment result. Technical Effects

[0007] The present invention reduces the attention calculation overhead by condensing the feature map through the representative token generation module; the window internal attention module allocates the attention weight within the window to the copied feature map; the overall framework realizes the combination of local information interaction and global information interaction. Compared with the prior art, the present invention greatly improves the accuracy of no-reference image evaluation (SRCC and PLOCC indicators); the feature extraction module in this technical means can be composed of various feature extraction backbones, and this framework can greatly improve the image quality evaluation performance of these backbone networks. In other words, this technical means has excellent generalization ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 is a system structure diagram of the present invention;

[0009] Figure 2 Schematic diagram of the global attention calculation module;

[0010] Figure 3 Schematic diagram of the attention module within the window;

[0011] Figure 4 Schematic diagram for visualizing the feature map evaluation. DETAILED DESCRIPTION

[0012] like Figure 1 As shown, this embodiment relates to a non-reference image quality assessment system based on concentrated attention, including: a feature extraction module, an adjustment module, a local spatial information encoding module, a representative token generation module, a global attention calculation module, a replication module, an in-window attention module and a quality assessment module, wherein: the feature extraction module extracts the original feature map from the input image The adjustment module performs pointwise convolution processing on the original feature map M and adjusts the channel dimension c to obtain the feature map The local spatial information encoding module uses depthwise convolution (convolution kernel size = 3×3, padding = 1, step size = 1) to encode the feature map M a Encode local spatial information to obtain the encoded feature map Complete local information interaction; the representative token generation module is based on the feature map M l Window division is performed, and the representative token (Outstand token) corresponding to each window is generated through deep convolution, that is, the representative (Outstanding) feature map in: win is the window size, which is also the convolution kernel size and step size; the global attention calculation module calculates self-attention based on the representative token to complete low-cost global attention interaction; the copy module copies the interaction result according to the window size corresponding to the token to obtain the feature map The window attention module is based on the feature map M a Perform point-by-point convolution to extract the private information of each element of the matrix Divide it into windows (win×win) and use the Softmax function on each window to get the weight matrix Quality assessment module for M w2 With M d Multiply and add M l The added results are subjected to regression calculation to obtain the quality assessment result.

[0013] The feature extraction module is implemented through a network backbone of a CNN or Transformer structure, such as SwinTransformer, ViT, ResNet, ConvNeXt, EfficientNet, etc.

[0014] The adjustment module is implemented by pointwise convolution technology, and the size of the convolution kernel is 1×1×C, where C is the number of channels of the previous layer. Pointwise convolution performs weighted combination of feature maps in the depth direction.

[0015] The local spatial information encoding module is implemented through depthwise convolution technology, and each channel of the input feature map is independently convolved, with one convolution kernel processing one channel.

[0016] The representative token generation module will Divide into several square windows of size win×win, and generate a token for each window through depthwise convolution with kernel size = win = step size. The tokens generated by all windows constitute the outstanding feature map

[0017] like Figure 2 As shown, the global attention calculation module represents the token feature map M o The spatial dimension is flattened and compressed to obtain the compressed feature map Where N = h′×w′, and then the compressed feature map F is transformed through three linear layers respectively. o , get Query, Key, Value projection, and divide each projection into hd group matrices Where hd is the number of heads of multi-head self-attention. To handle the permutation invariance of Multi-Head Self-Attention, relative position encoding is added in the process of calculating the attention of each head. Specifically: i The corresponding relative position offset matrix B i It can be regarded as a global static convolution kernel, which is added to the attention matrix: in: c′ is the number of channels after adjustment.

[0018] The global attention calculation module is further provided with layer normalization and residual connection structure to improve the training stability of the network and enhance the information flow.

[0019] The copy module is implemented using Python broadcast technology to copy the compressed feature map that completes the global attention calculation back to its original size.

[0020] like Figure 3 As shown, the window attention module is based on the feature map M a Perform point-by-point convolution to extract the private information of each element of the matrix Divide it into windows (win×win) and use the Softmax function on each window to get the weight matrix

[0021] The quality assessment module refers to: w2 First with M d Multiply and add M l The result of addition (M w2 With M d Multiply by to get personalized global information, andl After addition, personalized global information and local information are combined). After convolution and average pooling, the features are mapped to quality scores, and the training is performed with minimizing the mean absolute error (MAE) as the loss function. Specifically: Where: p i is the prediction score of the i-th image, s i is the corresponding true value, and N is the batch size.

[0022] After specific practical experiments, under the specific hardware environment settings of Pytorch 1.12.1, CUDA 11.6, and NVDIA RTX 3090GPU, the above method was run with the image cropped to 224×224 size, Batch_size=16, learning rate=3e-5, Weight-Decay=1e-5, and optimizer=Adam. The experimental results obtained are: SOTA is achieved on multiple standard IQA datasets and the performance is excellent after replacing the backbone, as follows: Dataset name Number of pictures Number of distortion types Distortion Type CSIQ 866 6 Synthetic TID 3000 24 Synthetic KADID 10125 25 Synthetic KONIQ 10073 - authentic LIVEChallenge 1162 - authentic

[0023] The evaluation criteria used are the PLCC and SROCC algorithms used for standard IQA problems, including: Where: N is the number of test images, and i are the predicted quality score and true quality score of the i-th image, respectively. and are the means of the first two, d i is the ranking difference between the predicted value and the true value of the i-th image.

[0024] The range of the PLCC and SROCC algorithms is between -1 and 1, and the larger the absolute value, the better the prediction performance of the method.

[0025] The following table shows the performance comparison of the PLCC and SROCC evaluation algorithms on five standard image quality assessment datasets, covering both artificial distortion and real distortion. This embodiment significantly surpasses all existing methods in CSIQ, TID, KADID, and CLIVE, and the results on KONIQ-10k are also close to the current best method TReS. It can be seen that this embodiment can make accurate judgments on both artificially synthesized distorted images and real distorted images in the wild, and has good generalization ability.

[0026] To prove that the present invention (Espresso) has good generalization ability on various CNN / Transformer structures, the performance of various network backbones after adding Espresso is compared on the CLIVE dataset, as shown in the following table: Model structure SROCCPLCC SwinT 0.8610.873 SwinT*+Espresso 0.8770.893 ViT 0.8570.874 ViT*+Espresso 0.8630.884 ConvNeXt 0.8290.848 ConvNeXt*+Espresso 0.8470.854 EfficientNet 0.7960.805 EfficientNet*+Espresso 0.8350.852 ResNet18 0.7640.771 ResNet18*+Espresso 0.8060.819

[0027] To further demonstrate the effectiveness, lightness, and versatility of the present invention, ResNet is used as the backbone network to compare the performance of the parameter quantity and accuracy with the mainstream convolution-Transformer combined architecture module. The results are as follows: Model structure Parameter quantity SROCCPLCC ResNet50 23,510,081 0.8430.864 ResNet* 8,544,321 0.8450.860 ResNet*+BoT 15,960,897 0.8560.874 ResNet*+MobileVit 18,039,105 0.8460.862 ResNet*+GhostNet 15,220,801 0.8210.836 ResNet*+Espresso 10,670,657 0.8630.882

[0028] It can be found that ResNet50*+Espresso has the least number of parameters and the best performance, which further proves the versatility of the present invention. In addition, compared with the extensive attention algorithms of BoT and MobileViT, the present invention can achieve good global information interaction by using only a small number of representative elements for attention operations.

[0029] Compared with the prior art, the complexity of calculating the self-attention of the corresponding window based on the representative token in the present invention is Compared with the existing BOT method that performs self-attention calculation on all elements of the feature map (Aravind Srinivas, Tsung-Yi Lin, Niki Parmar, Jonathon Shlens, Pieter Abbeel, and Ashish Vaswani. Bottleneck transformers for visual recognition. In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, pages 16519-16529, 2021), the computational complexity of the present invention is times; compared with the existing MobileViT method that performs self-attention calculation on all corresponding position elements in the window (SachinMehta and Mohammad Rastegari.Mobilevit:light-weight,general-purpose,andmobile-friendly vision transformer.arXiv preprint arXiv:2110.02178,2021), the computational complexity of the present invention is times.

[0030] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principle and purpose of the present invention. The protection scope of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. Each implementation scheme within its scope shall be subject to the constraints of the present invention.

Claims

1. A no-reference image quality assessment method based on concentrated attention, characterized in that: After extracting the features of the input image and encoding the local information to obtain the encoded feature map of local information interaction, the encoded feature map is divided into windows and a representative token is generated in each window. The representative token is used to replace the original complete feature map to perform multi-head self-attention calculation and obtain the updated feature map. The quality assessment result is obtained by performing regression calculation after the updated feature map is updated and then fused with the encoded feature map that interacts with the local information.

2. The method for no-reference image quality assessment based on concentrated attention according to claim 1, characterized in that: The encoded feature map of local information interaction is obtained by extracting the original feature map from the input image. After that, pointwise convolution is performed to adjust the channel dimension c to obtain the feature map For feature map M a Perform depthwise convolution to achieve local spatial information encoding and obtain the encoded feature map Complete local information interaction.

3. The method for no-reference image quality assessment based on concentrated attention according to claim 1, characterized in that: The generation of representative tokens refers to: according to the feature map M l Divide the window and generate the representative token corresponding to each window through deep convolution, that is, the representative feature map in: win is the window size, which is also the convolution kernel size and step size.

4. The method for no-reference image quality assessment based on concentrated attention according to claim 3, characterized in that: The updated feature map is obtained by calculating the self-attention based on the representative token, completing the low-cost global attention interaction, and then copying the interaction result according to the window size corresponding to the token to obtain the feature map Encoding feature map M for local information interaction a Perform point-by-point convolution to extract the private information of each element of the matrix Divide it into windows (win×win) and use the Softmax function on each window to get the weight matrix 5. The method for no-reference image quality assessment based on concentrated attention according to claim 3, characterized in that: The quality assessment result is obtained by performing regression calculation by adding the updated feature map after allocating attention within the window to the encoded feature map that interacts with the local information.

6. A non-reference image quality assessment system implementing the method of any one of claims 1 to 5, characterized in that: include: A feature extraction module, an adjustment module, a local spatial information encoding module, a representative token generation module, a global attention calculation module, a replication module, an in-window attention module and a quality assessment module, wherein: the feature extraction module extracts features from the input image; the adjustment module adjusts the channel dimension of the feature map; the local spatial information encoding module encodes local information on the feature map; the representative token generation module divides the feature map into windows and generates representative tokens for each window; the global attention calculation module calculates global self-attention in a low-cost manner based on the representative token; the replication module replicates the token according to the window size represented by the token; the in-window attention module allocates in-window attention weights to the feature map obtained after replication; the quality assessment module performs regression calculation based on the result of adding the feature map after attention allocation and the feature map after local information interaction to obtain a quality assessment result.

7. The no-reference image quality assessment system according to claim 6, characterized in that: The global attention calculation module represents the token feature map M o The spatial dimension is flattened and compressed to obtain the compressed feature map Where N = h'×w', and then the compressed feature map F is transformed through three linear layers respectively. o , get Query, Key, Value projection, and divide each projection into hd group matrices Where hd is the number of heads of multi-head self-attention. To handle the permutation invariance of Multi-Head Self Attention, relative position encoding is added in the process of calculating the attention of each head. Specifically: i The corresponding relative position offset matrix B i It can be regarded as a global static convolution kernel, which is added to the attention matrix: in: c' is the number of channels after adjustment.

8. The no-reference image quality assessment system according to claim 6, characterized in that: The global attention calculation module is further provided with layer normalization and residual connection structure to improve the training stability of the network and enhance the information flow.

9. The no-reference image quality assessment system according to claim 6, characterized in that: The quality assessment module refers to: w2 First with M d Multiply and add M l The result of the addition, M w2 With M d Multiply by to get personalized global information, and l After addition, the personalized global information and local information are combined. After convolution and average pooling, the features are mapped to quality scores, and the training is performed with minimizing the mean absolute error (MAE) as the loss function. Specifically: Where: p i is the prediction score of the i-th image, s i is the corresponding true value, and N is the batch size.

Images