No-reference image quality assessment method based on concentrated attention

Abstract

A no-reference image quality assessment method based on concentrated attention is proposed. In the offline stage, a no-reference image quality assessment system based on concentrated attention is constructed. Features of the input image are extracted, and after window partitioning and representative token extraction, a weight matrix corresponding to each window is obtained. This weight matrix is ​​then superimposed with features encoded with local spatial information to achieve the fusion of local and global information. After training, this system is used for image quality assessment in the online stage. This invention combines the advantages of convolution and attention mechanisms. By using window partitioning to generate representatives, it captures long-distance dependencies between elements with minimal computation. The attention module within each window allows each element to have global information added as needed.

Description

Technical Field

[0001] This invention relates to a technique in the field of image processing, specifically a no-reference image quality assessment method based on Espresso Attention (IQA). Background Technology

[0002] Existing networks such as Vision Transformer for image quality assessment can significantly improve the image evaluation performance of traditional convolutional models when faced with complex real-world distortions. However, due to the computational complexity of multi-head attention, Transformer networks often require more computational resources and inference time than CNNs. To better combine the advantages of CNNs and Transformers, many hybrid architectures have been proposed for image recognition, object detection, and semantic segmentation tasks. However, the performance improvement of these hybrid architecture models in image quality assessment remains very limited. Summary of the Invention

[0003] This invention addresses the problem that existing technologies cannot generalize to various backbone networks and require image preprocessing, which increases complexity. It proposes a no-reference image quality assessment method based on condensed attention. Combining the advantages of convolution and attention mechanisms, it uses window partitioning to generate representatives, capturing long-distance dependencies between elements with minimal computation. Through an in-window attention module, each element in the window can have global information added as needed. This method can be applied to different types of state-of-the-art CNN / Transformer network backbones and significantly improves the performance of network backbones in image quality assessment.

[0004] This invention is achieved through the following technical solution:

[0005] This invention relates to a no-reference image quality assessment method based on concentrated attention. The method involves extracting features from the input image and encoding local information to obtain an encoded feature map of local information interaction. The encoded feature map is then divided into windows, and a representative token is generated in each window. This representative token is used to replace the original complete feature map for multi-head self-attention computation, resulting in an updated feature map. The updated feature map is then fused with the encoded feature map of local information interaction after in-window attention allocation, and the quality assessment result is obtained through regression computation.

[0006] This invention relates to a referenceless image quality assessment system based on condensed attention, implementing the above-mentioned method. The system includes: a feature extraction module, an adjustment module, a local spatial information encoding module, a representative token generation module, a global attention calculation module, a copying module, an in-window attention module, and a quality assessment module. Specifically: the feature extraction module extracts features from the input image; the adjustment module adjusts the channel dimensions of the feature map; the local spatial information encoding module encodes local information in the feature map; the representative token generation module divides the feature map into windows and generates a representative token for each window; the global attention calculation module calculates global self-attention in a low-cost manner based on the representative tokens; the copying module copies the tokens according to the window size represented by the tokens; the in-window attention module assigns in-window attention weights to the copied feature map; and the quality assessment module performs regression calculations based on the sum of the attention-allocated feature map and the feature map after local information interaction to obtain the quality assessment result.

[0007] Technical effect

[0008] This invention reduces attention computation overhead by condensing feature maps through a token generation module; an in-window attention module assigns attention weights to the copied feature maps within the window; and the overall framework combines local and global information interaction. Compared to existing technologies, this invention significantly improves the accuracy of no-reference image evaluation (SRCC and PLOCC metrics). The feature extraction module in this technique can be composed of various feature extraction backbones, and this framework can significantly improve the image quality evaluation performance of these backbone networks; in other words, this technique has excellent generalization ability. Attached Figure Description

[0009] Figure 1 This is a system structure diagram of the present invention;

[0010] Figure 2 This is a schematic diagram of the global attention calculation module;

[0011] Figure 3 This is a schematic diagram of the attention module within the window;

[0012] Figure 4 This is a schematic diagram for visual evaluation of feature maps. Detailed Implementation

[0013] like Figure 1 As shown, this embodiment relates to a no-reference image quality assessment system based on condensed attention, including: a feature extraction module, an adjustment module, a local spatial information encoding module, a representative token generation module, a global attention calculation module, a copying module, an in-window attention module, and a quality assessment module, wherein: the feature extraction module extracts the original feature map from the input image. The adjustment module performs pointwise convolution on the original feature map M, adjusting the channel dimension c to obtain the feature map. The local spatial information encoding module processes the feature map M through depthwise convolution (kernel size = 3×3, padding = 1, stride = 1). a Local spatial information is encoded to obtain the encoded feature map. Complete partial information interaction; the token generation module represents the feature map M. l The window is divided, and a representative token (Outstanding token), i.e., an Outstanding feature map, is generated for each window through depthwise convolution. in: `win` represents the window size, as well as the convolution kernel size and stride; the global attention calculation module calculates self-attention based on the representative token, completing low-cost global attention interaction; the copying module copies the interaction result according to the window size corresponding to the token, obtaining the feature map. The in-window attention module is based on feature map M a Perform pointwise convolution to extract the private information matrix for each element. Divide it into windows (win×win) and apply the Softmax function to each window to obtain the weight matrix. The quality assessment module for M w2 With M d Multiply and then multiply by M l The summed results are then subjected to regression calculations to obtain the quality assessment results.

[0014] The feature extraction module is implemented through a network backbone of CNN or Transformer structure, such as SwingTransformer, ViT, ResNet, ConvNeXt, EfficientNet, etc.

[0015] The adjustment module is implemented using pointwise convolution, with a kernel size of 1×1×C, where C is the number of channels in the previous layer. Pointwise convolution performs a weighted combination of feature maps along the depth direction.

[0016] The local spatial information encoding module is implemented using depthwise convolution technology, which performs independent convolution calculations on each channel of the input feature map, with one convolution kernel processing one channel.

[0017] The aforementioned token generation module will The data is divided into several square windows of size win × win. For each window, a token is generated by depthwise convolution with kernel size = win = stride. The tokens generated from all windows constitute the outstanding feature map.

[0018] like Figure 2 As shown, the global attention calculation module calculates the representative token feature map M. o The compressed feature map is obtained by flattening and compressing the spatial dimension. Where N = h′ × w′, then the compressed feature map F is transformed through three linear layers respectively. o The query, key, and value projections are obtained, and each projection is divided into hd groups of matrices. Where hd represents the number of heads in the multi-head attention mechanism; to handle the permutation invariance of Multi-Head SelfAttention, relative position encoding is added during the calculation of the attention for each head. Specifically: for each attention head hd... i The corresponding relative position offset matrix B i It can be viewed as a global static convolution kernel, which is added to the attention matrix: in: c′ represents the adjusted number of channels.

[0019] The global attention calculation module further includes layer normalization and residual connection structures to improve the training stability of the network and enhance information flow.

[0020] The copy module is implemented using Python broadcasting technology to copy and restore the compressed feature map, which has completed the global attention calculation, to its original size.

[0021] like Figure 3 As shown, the in-window attention module is based on feature map M a Perform pointwise convolution to extract the private information matrix for each element. Divide it into windows (win×win) and apply the Softmax function to each window to obtain the weight matrix.

[0022] The aforementioned quality assessment module refers to: based on M w2 First with M d Multiply and then multiply by M l The result of the addition (M) w2 With M d Multiplying them yields personalized global information, which is then compared with M.l The summation achieves a combination of personalized global and local information. After convolution and average pooling, the features are mapped to quality scores, and training is performed using the minimum mean absolute error (MAE) as the loss function. Specifically: Where: p i Let s be the prediction score for the i-th image. i The corresponding actual value is N, and the batch size is N.

[0023] Through specific practical experiments, under the hardware environment settings of PyTorch 1.12.1, CUDA 11.6, and NVIDIA RTX 3090 GPU, with image cropping to 224×224 size, Batch_size=16, learning rate=3e-5, Weight-Decay=1e-5, and optimizer=Adam, the experimental results show that the method achieves state-of-the-art performance on multiple standard IQA datasets and exhibits excellent performance after backbone replacement, as detailed below:

[0024] Dataset name Number of images Number of Twist Types Twisted type CSIQ 866 6 Synthetic TID 3000 24 Synthetic KADID 10125 25 Synthetic KONIQ 10073 - authentic LIVEChallenge 1162 - authentic

[0025] The evaluation criteria used are the PLCC and SROCC algorithms used in standard IQA problems, specifically including: Where: N is the number of test images. and s i Let be the predicted quality score and the true quality score of the i-th image, respectively. and These are the means of the first two, d. i Let be the difference between the predicted and actual values ​​for the i-th image.

[0026] The range of the PLCC and SROCC algorithms is between -1 and 1, and the larger the absolute value, the better the prediction performance of the method.

[0027] The table below shows a performance comparison of the PLCC and SROCC evaluation algorithms on five standard image quality assessment datasets, covering both artificial and real-world distortion scenarios. This embodiment significantly outperforms all existing methods on CSIQ, TID, KADID, and CLIIVE, and its results on KONIQ-10k are close to the current state-of-the-art method, TReS. Therefore, this embodiment demonstrates good generalization ability, accurately judging both artificially synthesized distorted images and real-world distorted images.

[0028]

[0029]

[0030] To demonstrate that this invention (Espresso) has good generalization ability on various CNN / Transformer architectures, the performance of various network backbones with Espresso added was compared on the CLIVY dataset, as shown in the table below:

[0031] Model Structure SROCCPLCC SwinT 0.8610.873 SwinT*+Espresso 0.8770.893 ViT 0.8570.874 ViT*+Espresso 0.8630.884 ConvNeXt 0.8290.848 ConvNeXt*+Espresso 0.8470.854 EfficientNet 0.7960.805 EfficientNet*+Espresso 0.8350.852 ResNet18 0.7640.771 ResNet18*+Espresso 0.8060.819

[0032] To further demonstrate the effectiveness, lightweight nature, and versatility of this invention, ResNet was used as the backbone network, and its performance in terms of parameter count and accuracy was compared with mainstream convolutional-Transformer combined architecture modules. The results are as follows:

[0033] Model Structure Parameters SROCCPLCC ResNet50 23,510,081 0.8430.864 ResNet* 8,544,321 0.8450.860 ResNet*+BoT 15,960,897 0.8560.874 ResNet*+MobileVit 18,039,105 0.8460.862 ResNet*+GhostNet 15,220,801 0.8210.836 ResNet*+Espresso 10,670,657 0.8630.882

[0034] It can be observed that ResNet50*+Espresso has the fewest parameters and the best performance, further demonstrating the versatility of this invention. Furthermore, compared to the extensive attention algorithms of BoT and MobileViT, this invention achieves good global information interaction using only a small number of representative elements for attention operations.

[0035] Compared with existing technologies, the complexity of calculating the self-attention of the corresponding window based on the representative token in this invention is significantly reduced. Compared to existing BOT methods that perform self-attention computation on all elements of the feature map (Aravind Srinivas, Tsung-Yi Lin, Niki Parmar, Jonathon Shlens, Pieter Abbeel, and Ashish Vaswani. Bottleneck transformers for visual recognition. In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, pages 16519-16529, 2021), the computational cost of this invention is [amount missing]. The computational cost of this invention is times that of existing MobileViT methods that perform self-attention calculations on all corresponding elements within a window (SachinMehta and Mohammad Rastegari. Mobilevit: light-weight, general-purpose, and mobile-friendly vision transformer. arXiv preprint arXiv:2110.02178,2021). times.

[0036] The above-described specific implementations can be partially adjusted by those skilled in the art in different ways without departing from the principles and purpose of the present invention. The scope of protection of the present invention is defined by the claims and is not limited to the above-described specific implementations. All implementation schemes within the scope of the claims are bound by the present invention.

Claims

1. A no-reference image quality assessment method based on concentrated attention, characterized in that, After extracting features from the input image and encoding local information to obtain an encoded feature map of local information interaction, the encoded feature map is divided into windows and a representative token is generated in each window. The representative token is used to replace the original complete feature map to perform multi-head self-attention calculation and obtain an updated feature map. The quality assessment result is obtained by fusing the updated feature map with the encoded feature map that interacts with local information after applying in-window attention, and then performing regression calculation. The encoded feature map of the local information interaction is obtained by extracting the original feature map from the input image. Then, pointwise convolution is performed to process the channel dimensions. Adjustments are made to obtain the feature map. ; for feature maps Perform depthwise convolution to encode local spatial information and obtain encoded feature maps. To complete partial information interaction; The generation of representative tokens refers to: based on the feature map The window is divided, and a representative token, i.e., a representative feature map, is generated for each window through depthwise convolution. ,in: , This represents the window size, as well as the kernel size and stride. The updated feature map is obtained as follows: Self-attention is calculated based on the representative tokens; after completing a low-cost global attention interaction, the interaction result is copied according to the window size corresponding to the token to obtain the feature map. ; Encoding feature map of local information interaction Perform pointwise convolution to extract the private information matrix for each element. Press it in the window ( The data is divided into two parts, and the Softmax function is applied to each window to obtain a weight matrix. ; The quality assessment result is obtained by adding the updated feature map after in-window attention allocation to the encoded feature map of local information interaction and then performing regression calculation. The quality assessment module is based on First with Multiply and then add The result of addition, and Multiplying yields personalized global information, and... The summation combines personalized global and local information, and after convolution and average pooling, the features are mapped to quality scores.

2. A referenceless image quality assessment system for implementing the method of claim 1, characterized in that, include: The system comprises a feature extraction module, an adjustment module, a local spatial information encoding module, a representative token generation module, a global attention calculation module, a copying module, an in-window attention module, and a quality evaluation module. Specifically: the feature extraction module extracts features from the input image; the adjustment module adjusts the channel dimensions of the feature map; the local spatial information encoding module encodes local information in the feature map; the representative token generation module divides the feature map into windows and generates a representative token for each window; the global attention calculation module calculates global self-attention based on the representative tokens in a low-cost manner; the copying module copies the tokens according to the window size represented by the tokens; the in-window attention module assigns in-window attention weights to the copied feature map; and the quality evaluation module performs regression calculations based on the sum of the attention-allocated feature map and the feature map after local information interaction, yielding the quality evaluation result.

3. The referenceless image quality assessment system according to claim 2, characterized in that, The global attention calculation module represents the token feature map. The compressed feature map is obtained by flattening and compressing the spatial dimension. ,in Then, the compressed feature maps are transformed through three linear layers. Get the Query, Key, and Value projections, and divide each projection into... Group matrix ,in The number of heads in the multi-head attention mechanism; to handle the permutation invariance of Multi-Head Self Attention, relative position encoding is added during the calculation of the attention of each head. Specifically: each attention head Corresponding relative position offset matrix It can be viewed as a global static convolution kernel, which is added to the attention matrix: ,in: , This is the adjusted number of channels.

4. The referenceless image quality assessment system according to claim 2, characterized in that, The global attention calculation module further includes layer normalization and residual connection structures to improve the training stability of the network and enhance information flow.

5. The referenceless image quality assessment system according to claim 2, characterized in that, The quality assessment module is trained using the minimization of mean absolute error (MAE) as the loss function, specifically as follows: ,in: No. The predicted score of the image. The corresponding actual value, This refers to the batch size.

Images