A no-reference image quality assessment method fusing convolution and attention mechanism

Abstract

The application discloses a kind of no reference image quality evaluation methods of fusion convolution and attention mechanism, comprising: pre-processing to input image, obtain the image after processing;By adding the feature of the image after processing is extracted to double attention deep convolutional neural network, image feature is weighted, and based on global average pooling operation, the image feature is compressed, and the image quality score is predicted using full connection layer.The application uses deeper convolution network structure, feature extraction is more comprehensive;At the same time, deep convolution fusion double attention mechanism combination effective feature information, so as to improve image quality prediction accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image quality assessment, and in particular to a no-reference image quality assessment method fusing convolution and attention mechanism. BACKGROUND

[0002] Image quality assessment is a very practical evaluation technique. With the development of computer vision, more and more image tasks are conquered, and whether the image quality processed by the computer is improved or not becomes a new topic, and the field of image quality assessment follows. Image quality assessment can be divided into full-reference image quality assessment and no-reference image quality assessment. The full-reference method is to compare the image to be evaluated with its original reference image to calculate the difference measure between the two to evaluate the image quality. Commonly used evaluation indexes include peak signal-to-noise ratio, structural similarity index, etc. The no-reference method does not rely on the reference image and directly evaluates the quality of the image to be evaluated. The no-reference method mainly extracts and analyzes the local features and global features of the image to obtain the evaluation result of the image quality. Full-reference image quality assessment needs the original reference image as a comparison benchmark, so it is more limited in practical application. The no-reference method is more flexible in practical application and can be used for image quality assessment in various situations.

[0003] At present, some methods based on extracting image features (such as statistical information, brightness, contrast, clarity, color, etc.) have achieved good evaluation accuracy by fusing image processing, image analysis and human visual perception theory. However, this traditional method also faces the problem of being limited by the selection and calculation method of the features, so the no-reference image quality assessment method based on deep learning is more competitive. Deep learning method is a new no-reference image quality assessment method that has appeared in recent years, which is mainly based on the training and prediction of deep neural network model.

[0004] When extracting features, the traditional method uses a specified feature extraction method, which often needs to be targeted at a specific type of image. The deep learning method learns the image quality assessment model from a large amount of image data, optimizes the weights and structure of the model through network training, and thus obtains more accurate quality assessment results. Compared with the traditional method, the deep learning method does not need to manually select features and calculation methods, has better generalization ability and robustness, and can handle more complex and diverse image quality assessment tasks. SUMMARY

[0005] The application aims to provide a no-reference image quality assessment method fusing convolution and attention mechanism, which uses deep convolutional neural network to extract image features with characteristics of hierarchy, semantics and abstraction, fully mines and uses features at all levels in the image, combines human visual perception model to score and predict the image, uses the idea of double attention mechanism to optimize convolution results, strengthens the proportion of space and channel with major influence, and obtains more representative feature maps.

[0006] To achieve the above-mentioned purpose, the application provides a no-reference image quality assessment method fusing convolution and attention mechanism, comprising:

[0007] Pretreating the input image to obtain a processed image;

[0008] Extracting features of the processed image through a deep convolutional neural network with double attention, weighting the image features, compressing the image features based on global average pooling operation, and predicting image quality score using a full connection layer.

[0009] Preferably, the pretreatment of the input image comprises:

[0010] Adjusting the input image to the same size, randomly transforming the adjusted image, and performing data enhancement;

[0011] Adjusting the pixels of the image within a preset pixel range, and normalizing the image;

[0012] Finally, converting the type of the image into a data type suitable for network input, and completing the pretreatment of the input image.

[0013] Preferably, the deep convolutional neural network with double attention is used to extract features of the processed image, comprising:

[0014] Inputting the processed image into a deep ResNet convolutional network with double attention to extract features, and obtaining high-dimensional feature representation;

[0015] Wherein, the features before the last convolutional layer of the deep ResNet convolutional network with double attention are selected as the feature representation of the image.

[0016] Preferably, the features output by the last convolutional layer of the deep ResNet convolutional network with double attention are 4D tensors with a shape of (batch_size, num_channels, height, width), wherein batch_size is batch size, num_channels is channel number, height and width are the height and width of the feature map.

[0017] Wherein, a double attention module is added after each residual block of the deep convolutional neural network ResNet to form the deep ResNet convolutional network with double attention.

[0018] Preferably, the image features are weighted, including:

[0019] Adaptive feature weighting is performed based on the 4D tensor, and attention weighting is performed through a channel attention module and a spatial attention module respectively to obtain a weighted feature tensor.

[0020] Preferably, the process of attention weighting through the channel attention module and the spatial attention module respectively is:

[0021] The output of the residual block is taken as input to perform channel attention processing, then a fully connected layer is connected to perform feature transformation, and then a sigmoid activation function is used to obtain the attention weight of each channel;

[0022] The output result of the channel attention is multiplied by the input of the residual block to obtain a weighted input, which is input into the spatial attention module;

[0023] In the spatial attention module, a convolution layer is used to perform convolution operation on the weighted input to obtain a feature map, and the feature map is taken as input to perform channel attention processing again to obtain the attention weight of each spatial position;

[0024] The output results of the channel attention and the spatial attention are multiplied to obtain a feature map fused with channel and spatial attention, which is taken as the input of the next residual block.

[0025] Preferably, the method for compressing the image features based on the global average pooling operation is:

[0026]

[0027] Wherein, x is the input feature map, y is the output feature map, y i,j,k represents the pixel value of the position (i,j) on the kth channel of the output feature map, H and W are the height and width of the input feature map respectively, and s is the step length of the pooling layer.

[0028] Compared with the prior art, the present application has the following advantages and technical effects:

[0029] The present application uses a deep convolutional neural network to extract more levels of image features, uses a double attention mechanism, conforms to the human eye visual model, focuses on the area of human eye attention, and forms a more reasonable feature map, thereby improving the accuracy of the no-reference image quality evaluation score. BRIEF DESCRIPTION OF DRAWINGS

[0030] The accompanying drawings, which form a part of this application, are included to provide a further understanding of the application and are incorporated in and constitute a part of this application. The embodiments of this application, and their

[0031] Fig. 1 is a deep neural network RESNET-50 network schematic diagram in the embodiment of the application;

[0032] Fig. 2 is a deep neural network and double attention mechanism network fusion schematic diagram in the embodiment of the application; wherein, C is the feature dimension, H and W are the height and width of the input feature map respectively. DETAILED DESCRIPTION

[0033] It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0034] It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown herein.

[0035] As shown in Figs. 1-2 The present application proposes a method for evaluating the quality of a reference-free image by fusing convolution and attention mechanism, which specifically comprises:

[0036] Step one, pre-processing the input image. After reading the image, in order to enable the model to process images of different resolutions, the read image needs to be scaled to a uniform size of 256x256. Then some random transformations are performed on the scaled image, such as rotation, cropping, translation, flipping, contrast adjustment, etc. for data enhancement. Divide the pixel value of the image by 255 (or subtract the mean value and divide by the standard deviation), so that the pixel value of the image is between 0 and 1, and complete the normalization of the image. Finally, convert the data type of the image to floating point or integer to adapt to the data type of the network input.

[0037] Deep convolutional neural network feature extraction:

[0038] First, the input image is adjusted to a size of 224x224 using cropping, scaling, or sliding windows. The resized photo is preprocessed, including normalization and other operations. The preprocessed photo is input into the deep convolutional neural network ResNet, which is deeper than traditional convolutional neural networks. ResNet uses residual learning to make it easier for the network to learn deeper features. This means that the deep convolutional neural network ResNet can extract more representative and abstract image features. At the same time, ResNe adds a shortcut connection structure to the middle layer of the network, which can effectively avoid the problem of gradient vanishing and gradient explosion, thereby better preserving the information of image features, which enables it to extract more representative and distinguishable image features. Since ResNet has a total of 50 layers, the feature layer before the last convolutional layer is selected for extraction, which can obtain a 1x2048x7x7 dimensional feature vector.

[0039] Step two, extract image features with deep ResNet convolutional network with double attention. In this embodiment, the input image is passed through a deep ResNet convolutional network with double attention to obtain a high-dimensional feature representation. Typically, the features before the last convolutional layer of the deep ResNet are selected as the image feature representation, as these features can better capture the semantic information of the image. The last convolutional layer of the deep ResNet outputs a 4D tensor with a shape of (batch_size, num_channels, height, width), where batch_size is the batch size, num_channels is the number of channels, and height and width are the height and width of the feature map. Here, the batch_size is usually 1, because each image is processed individually.

[0040] Double attention mechanism:

[0041] Apply the double attention mechanism to the deep convolutional neural network ResNet, and add a double attention module after each residual block of the deep convolutional neural network ResNet.

[0042] Take the first residual block as an example:

[0043] First, the output of the residual block is taken as input and processed with channel attention. This embodiment can use a global average pooling layer to average pool the channels, then a fully connected layer to transform the features, and then a sigmoid activation function to get the attention weight of each channel. This process can be represented by equation (1):

[0044]

[0045] where xi,j denotes the feature vector in the i-th row and j-th column of the input feature map, H and W denote the height and width of the feature map, respectively, and denote two fully connected layers in the channel attention, δ denotes the ReLU activation function, σ denotes the sigmoid activation function, and α c denotes the attention weight of the c-th channel.

[0046] Next, the output of the channel attention is multiplied by the input of the residual block to obtain a weighted input. This weighted input is input into the spatial attention module.

[0047] In the spatial attention module, a 3x3 convolution layer is used to convolve the weighted input to obtain a feature map. Then, the feature map is input into the channel attention again to obtain the attention weight of each spatial position. This process can be represented by formula (2):

[0048]

[0049] where z i,j denotes the feature vector in the i-th row and j-th column of the output feature map of the spatial convolution, and denote two fully connected layers in the spatial attention, β i,j denotes the attention weight of the (i, j) spatial position.

[0050] Finally, the output of the channel attention and the spatial attention is multiplied to obtain a feature map that integrates the channel and spatial attention, which is input into the next residual block.

[0051] The above is the design of the double attention module of the first residual block. Similarly, the present application can add a double attention module after the second convolution layer of the second, third, and fourth residual blocks of the deep convolutional neural network ResNet. The structures of these residual blocks are similar to the first residual block, but have different numbers of layers and convolution kernels.

[0052] Step three: Double attention mechanism is used for image feature weighting. The 4D tensor obtained by step two is used for adaptive feature weighting. Attention weighting is performed through the channel attention module and the spatial attention module respectively to obtain a weighted feature tensor.

[0053] Feature compression:

[0054] The present embodiment uses global average pooling for feature compression:

[0055]

[0056] where x is the input feature map, y is the output feature map, H and W are the height and width of the input feature map, respectively, and s is the step size of the pooling layer. In global average pooling, the size of the pooling window is the same as the size of the input feature map, i.e., H = W = height = width. i,j,k represents the pixel value at position (i, j) on the kth channel of the input feature map, y i,j,k represents the pixel value at position (i, j) on the kth channel of the output feature map.

[0057] The output dimension of the deep convolutional neural network ResNet network with double attention is 1 x 2048 x 7 x 7 = 100352. Therefore, global average pooling can reduce the network output dimension to 2048 for subsequent score prediction.

[0058] Step four, principal component analysis method is used for dimension reduction, and a fully connected layer is used to predict the image quality score.

[0059] The principal component analysis method includes five steps: data standardization, calculation of the covariance matrix, calculation of the eigenvalues and eigenvectors, selection of principal components, and construction of a new feature space. Finally, the new features after principal component analysis are put into a fully connected layer to predict the image score, and a trained image quality score model is used to evaluate the quality of the new image. The new image is input into the neural network, and a fully connected layer is used to predict the image score to judge the quality of the image.

[0060] In order to better enhance feature expression, the present application considers an attention mechanism that simulates human cognitive acuity, so that the multi-channel selection of features is more intelligent. This method extracts high-level features of images through a deep convolutional neural network, then uses a double attention mechanism to capture important local regions in the image, and fuses these local information to evaluate the quality of the image.

[0061] The above is only a preferred specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any changes or replacements within the technical scope disclosed in the present application can be easily thought of by those skilled in the art, and should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A no-reference image quality assessment method integrating convolution and attention mechanisms, characterized in that, include: Preprocess the input image to obtain the processed image; The image features are extracted by adding a deep convolutional neural network with dual attention, the image features are weighted, and the image features are compressed based on global average pooling. The image quality score is predicted using a fully connected layer. Features of the processed image are extracted using a deep convolutional neural network with dual attention, including: The processed image is then input into a deep ResNet convolutional network with dual attention for feature extraction to obtain a high-dimensional feature representation. In this context, features prior to the last convolutional layer of the deep ResNet convolutional network with dual attention are selected as the feature representation of the image. The last convolutional layer of the deep ResNet convolutional network with dual attention outputs a 4D tensor with a shape of (batch_size, num_channels, height, width), where batch_size is the batch size, num_channels is the number of channels, and height and width are the height and width of the feature map. Specifically, a deep ResNet convolutional network with added dual attention is formed by adding a dual attention module after each residual block of the deep convolutional neural network ResNet. The image features are weighted, including: Based on the 4D tensor, adaptive feature weighting is performed, and attention weighting is performed through the channel attention module and the spatial attention module respectively to obtain the weighted feature tensor; The process of attention weighting through the channel attention module and the spatial attention module respectively is as follows: The output of the residual block is used as input for channel attention processing; then it is connected to a fully connected layer for feature transformation, and then... The activation function obtains the attention weights for each channel, specifically: ； in, The first feature in the input feature map Line 1 eigenvectors of columns and These represent the height and width of the feature map, respectively. and This represents two fully connected layers in channel attention. express Activation function express Activation function Indicates the first Attention weights for each channel; The output of the channel attention is multiplied by the input of the residual block to obtain a weighted input, which is then input into the spatial attention module. In the spatial attention module, a convolutional layer is used to perform a convolution operation on the weighted input to obtain a feature map. This feature map is then used as input for channel attention processing to obtain the attention weights for each spatial location. Specifically: ； in, The first element in the output feature map of spatial convolution represents the... Line 1 eigenvectors of columns and This represents two fully connected layers in spatial attention. Indicates the first Attention weights for each spatial location; The outputs of channel attention and spatial attention are multiplied to obtain a feature map that fuses channel and spatial attention, which is then used as the input to the next residual block. The method for compressing the image features based on global average pooling is as follows: in, For the input feature map, To output the feature map, This indicates that the position on the k-th channel of the output feature map is... pixel values, and These represent the height and width of the input feature map, respectively. This represents the step size of the pooling layer.

2. The no-reference image quality assessment method fusing convolution and attention mechanisms according to claim 1, characterized in that, Preprocessing the input image includes: The input images are resized to the same size, and then random transformations are applied to the resized images to perform data augmentation. Adjust the image's pixels to a preset pixel range and then normalize the image. Finally, the image type is converted into a data type that adapts to the network input, completing the preprocessing operation of the input image.

Images