A blind image quality evaluation method based on complex-valued deep convolutional network and gradient information

Abstract

The application discloses a blind image quality evaluation method based on a complex deep convolution network and gradient information, and comprises the following steps: obtaining a distorted image to be evaluated; generating a corresponding gradient image by using a gradient detection operator; and respectively passing the distorted image and the corresponding gradient image through a complex deep convolution network with the same structure to extract multi-level features; wherein the complex deep convolution network firstly obtains high-frequency and low-frequency complex responses of the input image by using a dual-tree complex wavelet transform, then uses three layers of complex convolution blocks to extract multi-layer complex convolution features for the high-frequency and low-frequency responses, and uses a high-low-frequency feature aggregation block to fuse corresponding layer complex convolution features of the high-frequency branch and the low-frequency branch, and the fusion features of the high-low-frequency feature aggregation block are transmitted to a next layer high-low-frequency feature aggregation block to obtain multi-level features; and finally, the multi-level features output by the complex deep convolution network are cascaded for the distorted image and the gradient image. The application better extracts structural information.

Description

TECHNICAL FIELD

[0001] The application relates to a blind image quality evaluation method based on a complex-valued deep convolution network and gradient information, and belongs to the technical field of image quality evaluation. BACKGROUND

[0002] Image quality evaluation has important applications in the fields of image compression, image enhancement, etc. Image quality evaluation methods are generally divided into traditional methods based on manual feature extraction and methods based on deep learning. Current mainstream deep learning-based image quality evaluation methods mainly extract features through real-valued convolutional neural networks, and then complete the quality evaluation task. Research shows that complex-valued neural networks have more advantages in learning the phase and structural information of image signals.

[0003] Using a complex-valued deep convolution network to extract multi-level features of a distorted image, and combining gradient information to effectively express structural features for quality evaluation, better image quality evaluation results are expected. SUMMARY

[0004] In order to overcome the defects of the prior art, the application provides a blind image quality evaluation method based on a complex-valued deep convolution network and gradient information. The method explicitly uses gradient information of a distorted image, and uses a complex-valued deep convolution network with the same structure to extract multi-level features from the distorted image and a gradient image, and uses a high-low frequency feature aggregation block to fuse corresponding layer complex-valued convolution features of high-frequency branches and low-frequency branches. The method better extracts phase information and multi-scale structural information of the distorted image, thereby improving the accuracy of blind image quality evaluation.

[0005] To achieve the above purpose, the application adopts the following technical scheme:

[0006] The application provides a blind image quality evaluation method based on a complex-valued deep convolution network and gradient information, which comprises the following steps:

[0007] obtaining a distorted image to be evaluated;

[0008] generating a corresponding gradient image using a gradient detection operator;

[0009] extracting multi-level features from the distorted image and the corresponding gradient image through a complex-valued deep convolution network with the same structure; wherein the complex-valued deep convolution network first obtains high-frequency and low-frequency complex-valued responses of the input image using a dual-tree complex wavelet transform, then uses three layers of complex-valued convolution blocks to extract multi-layer complex-valued convolution features from the high-frequency and low-frequency responses, and uses a high-low frequency feature aggregation block to fuse corresponding layer complex-valued convolution features of high-frequency branches and low-frequency branches, and the fusion features of the high-low frequency feature aggregation block are transmitted to the next layer of high-low frequency feature aggregation blocks to obtain multi-level features.

[0010] Finally, the multi-level features of the distorted image and the gradient image output by the complex-valued deep convolutional network are cascaded and input into the quality prediction module to obtain the quality prediction score.

[0011] Furthermore, the method of generating a corresponding gradient image using a gradient detection operator to explicitly extract structural features of the distorted image includes:

[0012] The gradient image can be obtained using common gradient detection operators, such as the Sobel operator.

[0013] Furthermore, the distorted image and its corresponding gradient image are respectively passed through a complex-valued deep convolutional network with the same structure to extract multi-level features, including:

[0014] The complex-valued deep convolutional network first uses the dual-tree complex wavelet transform to obtain the high-frequency and low-frequency complex-valued responses of the input image, and then uses three layers of complex-valued convolution blocks to extract multi-layer complex-valued convolution features for both high-frequency and low-frequency responses. The high- and low-frequency feature aggregation block is used to fuse the corresponding layer complex-valued convolution features of the high-frequency branch and the low-frequency branch, and the fused features of the high- and low-frequency feature aggregation block are passed to the next layer of high- and low-frequency feature aggregation block to obtain multi-level features.

[0015] Furthermore, the complex-valued deep convolutional network first obtains high-frequency and low-frequency complex-valued responses of the input image using a dual-tree complex wavelet transform, including:

[0016] The distorted image and gradient image are decomposed by dual-tree complex wavelet transform to obtain the complex value response of the distorted image (including high-frequency features F H With the low frequency feature F L ) and the complex-valued response of the gradient image (including high-frequency features G H With the low-frequency feature G L ).

[0017] Furthermore, the three-layer complex-valued convolution block is used to extract multi-layer complex-valued convolution features for both high-frequency and low-frequency responses, including:

[0018] Three layers of complex-valued convolution blocks are used to extract multi-layer complex-valued convolution features for the high-frequency and low-frequency responses of the distorted image. and They represent the nth layer of complex-valued convolution features extracted from the high-frequency and low-frequency branches of the distorted image, and the corresponding feature dimensions are H and W represent the height and width of the input image, respectively. n Represents the number of channels for extracting features in the corresponding layer; similarly, three layers of complex-valued convolution blocks are used to extract multi-layer complex-valued convolution features for the high-frequency and low-frequency responses of the gradient image, respectively, which can be expressed as and

[0019] wherein each layer of complex-valued convolutional block is composed of three complex-valued convolutional operations (CV-conv3×3) with a kernel size of 3×3, and a complex-valued average pooling (AP) is used between the second and third complex-valued convolutional operations to reduce the spatial dimension. Taking the complex-valued convolutional feature extraction of the high-frequency and low-frequency branches of the distortion image as an example, the expression of the feature extraction is as follows:

[0020]

[0021] wherein n represents the index of the complex-valued convolutional block; respectively represent the output features of the nth complex-valued convolutional block of the high-frequency and low-frequency branches, and the corresponding feature dimensions are H and W represent the height and width of the distortion image, respectively, and C n represents the number of channels of the extracted features; CV-conv3×3 represents a 3×3 complex-valued convolution; and AP represents a complex-valued average pooling.

[0022] Further, the high-low frequency feature aggregation block is used to fuse the corresponding layer complex-valued convolutional features of the high-frequency branch and the low-frequency branch, and the fusion features of the high-low frequency feature aggregation block are transmitted to the next layer of high-low frequency feature aggregation block to obtain multi-level features, including:

[0023] The high-low frequency feature aggregation block is composed of complex-valued dilated convolution with different dilation rates and a channel attention module. The input (X n ) of the high-low frequency feature aggregation block is obtained by concatenating the high-frequency features the low-frequency features extracted by the corresponding layer complex-valued convolutional block, and the output features (F n-1 ) of the last layer of high-low frequency feature aggregation block. The input feature X n is first captured by complex-valued dilated convolution with different receptive field sizes to capture multi-scale context information, then the channel enhanced features are obtained by the channel attention module, and finally the output features of the current high-low frequency feature aggregation block are obtained by concatenating the channel enhanced features of different scales. The calculation process of the high-low frequency feature aggregation block can be expressed by the following formula:

[0024]

[0025] wherein AP represents a complex-valued average pooling; n represents the index of the high-low frequency feature aggregation block; and r represents the dilation rate; represents the weighted feature of the input feature X n after the dilated convolution with the dilation rate of r and through the channel attention module; GAP represents a complex-valued global average pooling; f c represents a 1×1 complex-valued convolution; σ represents a complex-valued Sigmoid activation; and δ represents a complex-valued ReLU activation; represents element-level multiplication; and fr represents a complex-valued dilation convolution with dilation rate r; represents a channel-wise concatenation; F n represents the output feature of the current high-low frequency feature aggregation block, and the output F of the third layer (index n = 3) high-low frequency feature aggregation block 3 as the multi-level features output by the complex-valued deep convolutional network for the input distorted image; similarly, the output G of the third layer (index n = 3) high-low frequency feature aggregation block can be obtained 3 as the multi-level features output by the complex-valued deep convolutional network for the input gradient image.

[0026] Further, the multi-level features output by the complex-valued deep convolutional network for the distorted image and the gradient image are concatenated and input to a quality prediction module to obtain a quality prediction score, which comprises:

[0027] The quality prediction module uses global average pooling and a multi-layer perceptron (MLP) containing three fully connected layers to fuse the above features and predict the perceptual quality. The predicted quality score is expressed as:

[0028]

[0029] where Q i is the quality prediction score of the i-th distorted image, F 3 and G 3 are the multi-level features extracted by the complex-valued deep convolutional network for the distorted image and the corresponding gradient image, respectively.

[0030] Further, the training method of the overall network model is as follows:

[0031] obtain training images;

[0032] generate gradient images corresponding to the training images using a gradient detection operator;

[0033] extract multi-level features from the training images and the corresponding gradient images through complex-valued deep convolutional networks with the same structure; concatenate the multi-level features output by the complex-valued deep convolutional network for the distorted image and the gradient image, and input them to a quality prediction module to obtain a quality prediction score;

[0034] According to the overall loss function of the complex-valued deep convolutional network image quality evaluation model, adjust the parameters of the complex-valued deep convolutional network model through gradient backpropagation to obtain the trained complex-valued deep convolutional network image quality prediction model.

[0035] Further, the overall loss function is the average absolute error between the quality scores predicted by the model and the subjective quality scores for all training images, and the overall loss function is expressed as follows:

[0036]

[0037] wherein Q i is the predicted quality score of the i-th image, s i is its corresponding subjective quality score, and N is the number of training images.

[0038] Compared with the prior art, the present application has the following beneficial effects:

[0039] 1. The present application provides a blind image quality evaluation method based on complex-valued deep convolutional network and gradient information, which better extracts structural information by using a double-branch network structure of distorted images and gradient images.

[0040] 2. The present application uses a complex-valued deep convolutional network to better express quality-related phase information than a real-valued convolutional network, and uses a high-low frequency feature aggregation block to better extract multi-scale semantic information, thus obtaining better image quality evaluation performance. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 is the overall flowchart of the blind image quality evaluation method based on complex-valued deep convolutional network and gradient information of the embodiment of the present application.

[0042] Figure 2 is a structural schematic diagram of the complex-valued deep convolutional network of the embodiment of the present application.

[0043] Figure 3 is a structural schematic diagram of the complex-valued convolutional block of the embodiment of the present application.

[0044] Figure 4 is a structural schematic diagram of the high-low frequency feature aggregation block of the embodiment of the present application. DETAILED DESCRIPTION

[0045] The present application will be further described below in combination with the drawings and embodiments. The following embodiments are only used to more clearly illustrate the technical solutions of the present application, and cannot be used to limit the protection scope of the present application.

[0046] The present application provides a blind image quality evaluation method based on complex-valued deep convolutional network and gradient information, comprising:

[0047] obtaining a distorted image to be evaluated;

[0048] generating a corresponding gradient image by using a gradient detection operator;

[0049] The distorted image and its corresponding gradient image are respectively passed through a complex-valued deep convolutional network with the same structure to extract multi-level features; the complex-valued deep convolutional network first uses the dual-tree complex wavelet transform to obtain the high-frequency and low-frequency complex-valued responses of the input image, and then uses three layers of complex-valued convolution blocks to extract multi-layer complex-valued convolution features for both high-frequency and low-frequency responses, and uses a high- and low-frequency feature aggregation block to fuse the corresponding layer complex-valued convolution features of the high-frequency branch and the low-frequency branch, and the fused features of the high- and low-frequency feature aggregation block are passed to the next layer of high- and low-frequency feature aggregation blocks to obtain multi-level features;

[0050] Finally, the multi-level features of the distorted image and the gradient image output by the complex-valued deep convolutional network are cascaded and input into the quality prediction module to obtain the quality prediction score.

[0051] This embodiment provides a blind image quality assessment method based on complex-valued deep convolutional networks and gradient information, such as Figure 1 As shown, the application process includes the following steps:

[0052] Step 1) Obtain the distorted image to be evaluated.

[0053] Step 2) Use the gradient detection operator to generate the corresponding gradient image.

[0054] In the implementation of the present invention, the Sobel horizontal and vertical gradient operators are used to perform template operations on the pixel area at the corresponding position of the distorted image to calculate the horizontal and vertical gradients of each pixel position of the distorted image. The Sobel horizontal and vertical gradient operators used (g x ,g y )as follows:

[0055]

[0056] The horizontal and vertical gradients of the distorted image I calculated at the pixel coordinate (i, j) are and Then the gradient amplitude (G) of the distorted image at position (i, j) is calculated according to the following formula: i，j ), the entire gradient image corresponding to the distorted image is recorded as G.

[0057]

[0058] Step 3) The distorted image and its corresponding gradient image are respectively passed through the complex-valued deep convolutional network with the same structure to extract multi-level features, such as Figure 2 First, the distorted image I and the gradient image G are decomposed by dual-tree complex wavelet transform, and the complex value response of the distorted image (including high-frequency features F H With the low frequency feature F L ) and the complex-valued response of the gradient image (including high-frequency features GH with low-frequency features G L ).

[0059] Step 4) Multiple complex-valued convolution features are extracted from the high-frequency and low-frequency responses of the distorted image and gradient image respectively using three-layer complex-valued convolution blocks, each of which is composed of three complex-valued convolution operations (CV-conv3x3) with a kernel size of 3x3, and a complex-valued average pooling (AP) is used between the second and third complex-valued convolutions to reduce the spatial dimension, as shown in Figure 3 . Taking the complex-valued convolution feature extraction of the high-frequency and low-frequency branches of the distorted image as an example, the expression of feature extraction is as follows:

[0060]

[0061] where n represents the index of the complex-valued convolution block; and represent the output features of the nth complex-valued convolution block of the high-frequency and low-frequency branches of the distorted image respectively, and the corresponding feature dimensions are H and W represent the height and width of the distorted image respectively, and C n represents the number of extracted features, taking values of {C n |64, 256, 512} respectively; CV-conv3x3 represents 3x3 complex-valued convolution; and AP represents complex-valued average pooling.

[0062] Similarly, multiple complex-valued convolution features are extracted from the high-frequency and low-frequency responses of the gradient image respectively using three-layer complex-valued convolution blocks, represented as and

[0063] Step 5) A high-low frequency feature aggregation block is used to fuse the corresponding layer complex-valued convolution features of the high-frequency branch and the low-frequency branch, as shown in Figure 4 , and the fusion features of the high-low frequency feature aggregation block are passed to the next layer of high-low frequency feature aggregation block to obtain multi-level features.

[0064] The high-low frequency feature aggregation block is composed of complex-valued dilated convolutions with different dilation rates and channel attention modules. The input (X n ) of the high-low frequency feature aggregation block is obtained by concatenating the high-frequency features and the low-frequency features extracted by the corresponding layer complex-valued convolution block, and the output features (F n-1 ) of the previous layer of high-low frequency feature aggregation block. The input features X n are first captured by complex-valued dilated convolutions with different receptive field sizes to capture multi-scale context information, then pass through the channel attention module to obtain channel-enhanced features, and finally concatenate the features of different scales after channel enhancement to obtain the output features of the current high-low frequency feature aggregation block. The calculation process of the high-low frequency feature aggregation block can be expressed by the following formula:

[0065]

[0066]

[0067] where AP denotes complex-valued average pooling; n denotes the index of the high-low frequency feature aggregation block; r denotes the dilation rate; denotes the input feature X n denotes the weighted feature after dilation convolution with a dilation rate of r and passing through the channel attention module; GAP denotes complex-valued global average pooling; f c denotes 1x1 complex-valued convolution; σ denotes complex-valued Sigmoid activation; δ denotes complex-valued ReLU activation; denotes element-wise multiplication; f r denotes complex-valued dilation convolution with a dilation rate of r; denotes channel dimension concatenation; F n denotes the output feature of the current high-low frequency feature aggregation block, the output F 3 of the third layer (index n = 3) high-low frequency feature aggregation block; similarly, the output G 3 of the third layer (index n = 3) high-low frequency feature aggregation block can be obtained.

[0068] where the specific process of the channel attention operation is as follows: first, the input feature X n is subjected to dilation convolution with a dilation rate of r to obtain f r (X n ), then f r (X n ) is subjected to complex-valued global average pooling operation along the channel dimension, the obtained result is subjected to 1x1 complex-valued convolution and complex-valued ReLU activation, and then subjected to 1x1 complex-valued convolution and complex-valued Sigmoid activation to obtain the corresponding channel dimension attention weight, which is multiplied with f r (X n ) element-wise to obtain the channel attention optimized feature.

[0069] Step 6) The multi-level features output by the complex-valued deep convolutional network from the distorted image and the gradient image are concatenated and input into the quality prediction module to obtain the quality prediction score. The quality prediction module uses global average pooling and a multi-layer perceptron (MLP) containing three fully connected layers to fuse the above features and predict the perceptual quality. The predicted quality score is expressed as:

[0070]

[0071] where Q iF is the quality prediction score of the i-th distorted image 3 and G 3 are multi-level features extracted by complex-valued deep convolutional networks respectively from the distorted image and the corresponding gradient image.

[0072] Compared with the general image quality evaluation method based on real-valued deep network, the blind image quality evaluation method based on complex-valued deep convolutional network and gradient information provided by the application makes better use of the quality-related phase information and structural information, and further improves the accuracy of image quality evaluation.

[0073] Those skilled in the art will understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0074] The present application is described with reference to flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus generate a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 means for performing the functions specified in the flowchart

[0075] These computer program instructions can also be stored in a computer-readable memory capable of causing a computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including instruction means, which implements the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 means for performing the functions specified in the flowchart

[0076] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus, so that a series of operation steps are performed on the computer or other programmable data processing apparatus to produce a computer-implemented process, so that the instructions executed on the computer or other programmable data processing apparatus provide a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1steps of the functions specified in the block or blocks.

[0077] The above description is merely that of the preferred embodiments of the application, and it is to be understood that numerous improvements and modifications will occur to those skilled in the art, and it is intended that the following claims cover all such improvements and modifications as fall within the scope of the application.

Claims

1. A blind image quality assessment method based on complex-valued deep convolutional networks and gradient information, characterized in that: include: Obtain a distorted image to be evaluated; Use the gradient detection operator to generate the corresponding gradient image; The distorted image and its corresponding gradient image are passed through a complex-valued deep convolutional network with the same structure to extract multi-level features; Finally, the distorted image and the gradient image are concatenated using the multi-level features output by the complex-valued deep convolutional network and input into the quality prediction module to obtain the quality prediction score. The distorted image and its corresponding gradient image are respectively passed through a complex-valued deep convolutional network with the same structure to extract multi-level features, including: The complex-valued deep convolutional network first uses the dual-tree complex wavelet transform to obtain the high-frequency and low-frequency complex-valued responses of the input image. Then, three layers of complex-valued convolution blocks are used to extract multi-layer complex-valued convolution features for both high-frequency and low-frequency responses. The high- and low-frequency feature aggregation block is used to fuse the corresponding layer complex-valued convolution features of the high-frequency branch and the low-frequency branch. The fused features of the high- and low-frequency feature aggregation block are passed to the next layer of high- and low-frequency feature aggregation blocks to obtain multi-level features. The three-layer complex-valued convolution block is used to extract multi-layer complex-valued convolution features for both high-frequency and low-frequency responses, including: Three layers of complex-valued convolution blocks are used to extract multi-layer complex-valued convolution features for the high-frequency and low-frequency responses of the distorted image. and They represent the nth layer of complex-valued convolution features extracted from the high-frequency and low-frequency branches of the distorted image, and the corresponding feature dimensions are H and W represent the height and width of the input image, respectively. n Represents the number of channels for extracting features in the corresponding layer; similarly, three layers of complex-valued convolution blocks are used to extract multi-layer complex-valued convolution features for the high-frequency and low-frequency responses of the gradient image, respectively, which can be expressed as and The high- and low-frequency feature aggregation block is used to fuse the corresponding layer complex-valued convolution features of the high-frequency branch and the low-frequency branch, and the fused features of the high- and low-frequency feature aggregation block are passed to the next layer of high- and low-frequency feature aggregation blocks to obtain multi-level features, including: The high- and low-frequency feature aggregation block is composed of complex-valued dilated convolutions with different dilation rates and channel attention modules. The input of the high- and low-frequency feature aggregation block is X n High-frequency features extracted by the complex-valued convolutional block of the corresponding layer Low-frequency characteristics And the output feature F of the high and low frequency feature aggregation block of the previous layer n-1 Cascaded, input feature X n First, multi-scale context information is captured through complex-valued dilated convolutions with different receptive field sizes. Then, channel-enhanced features are obtained through the channel attention module. Finally, the enhanced features of different scale channels are cascaded to obtain the output features of the current high- and low-frequency feature aggregation block. The calculation process of the high- and low-frequency feature aggregation block can be expressed by the following formula: Where AP represents complex average pooling; n represents the index of the high- and low-frequency feature aggregation block; r represents the expansion rate; Represents the input feature X n The weighted features after dilated convolution with dilation rate r and passing through the channel attention module; GAP represents complex-valued global average pooling; f c represents 1×1 complex-valued convolution; σ represents complex-valued Sigmoid activation; δ represents complex-valued ReLU activation; Indicates element-wise multiplication; f r represents a complex-valued dilated convolution with a dilation rate of r; represents channel dimension cascade; F n Represents the output features of the current high- and low-frequency feature aggregation block, the output F of the high- and low-frequency feature aggregation block of the third layer (index n=3) 3 As the multi-level features of the complex-valued deep convolutional network output of the input distorted image; similarly, the output G of the high- and low-frequency feature aggregation block of the third layer (index n=3) can be obtained 3 As the multi-level features output by the complex-valued deep convolutional network of the input gradient image.

2. The blind image quality assessment method based on complex-valued deep convolutional networks and gradient information according to claim 1, characterized in that: The method of generating a corresponding gradient image by using a gradient detection operator to explicitly extract structural features of the distorted image includes: The gradient image is obtained using a common gradient detection operator, namely the Sobel operator.

3. The blind image quality assessment method based on complex-valued deep convolutional networks and gradient information according to claim 1, characterized in that: The complex-valued deep convolutional network first uses the dual-tree complex wavelet transform to obtain the high-frequency and low-frequency complex-valued responses of the input image, including: The distorted image and the gradient image are decomposed by dual-tree complex wavelet transform to obtain a complex-valued response of the distorted image and a complex-valued response of the gradient image, respectively. The complex-valued response includes a high-frequency feature FH and a low-frequency feature FL, and the complex-valued response of the gradient image includes a high-frequency feature GH and a low-frequency feature GL.

4. The blind image quality assessment method based on complex-valued deep convolutional networks and gradient information according to claim 1, characterized in that: The overall loss function is the mean absolute error between the quality scores predicted by the model and the subjective quality scores of all training images. The overall loss function expression is as follows: Among them, Q i is the predicted quality score of the i-th image, s i is its corresponding subjective quality score, and N is the number of training images.

Images