Image quality assessment method based on edge detection and deep residual network

Abstract

The present invention discloses an image quality assessment method based on edge detection and deep residual network, which is specifically implemented in the following steps: Step 1, input the original distorted image, use the Canny edge detection algorithm to perform edge detection, and divide it into image blocks of the same size, and screen these image blocks; Step 2, normalize the screened image blocks, input them into the residual network, and output the image block scores and weights of the original distorted image; Step 3, perform weighted processing on the image block scores of the output original distorted image to obtain the predicted value of the image score. The present invention solves the problem of insufficient training data in traditional deep learning in the prior art, and at the same time uses the deep residual network to have a better extraction of shallow and deep features of the image.

Description

Technical Field

[0001] The present invention belongs to the technical field of image quality assessment, and in particular relates to an image quality assessment method based on edge detection and a deep residual network. Background Art

[0002] Images are an important source of information for human perception and machine pattern recognition, and their quality plays a decisive role in the adequacy and accuracy of the information obtained. However, images will inevitably be distorted to a certain extent during the process of acquisition, compression, processing, transmission, and display. In order to measure the quality of images and assess whether they meet certain application requirements, it is necessary to establish an image quality evaluation mechanism. Therefore, the research on image quality evaluation has also received widespread attention in recent years. Summary of the invention

[0003] The purpose of the present invention is to provide an image quality assessment method based on edge detection and deep residual network, which solves the problem of insufficient training data in traditional deep learning in the prior art, and at the same time utilizes the deep residual network to better extract shallow and deep features of the image.

[0004] The technical solution adopted by the present invention is an image quality assessment method based on edge detection and deep residual network, which is specifically implemented according to the following steps:

[0005] Step 1: Input the original distorted image, use the Canny edge detection algorithm to perform edge detection, and divide it into image blocks of the same size, and screen these image blocks;

[0006] Step 2: Normalize the filtered image blocks, input them into the residual network, and output the image block scores and weights of the original distorted image;

[0007] Step 3: Perform weighted processing on the image block scores of the output original distorted image to obtain a predicted value of the image score.

[0008] The present invention is also characterized in that:

[0009] Step 1 is as follows:

[0010] Step 1.1: First, crop the original distorted image to obtain image blocks of 32×32 pixels of the same size, label these image blocks, and output the image blocks with serial numbers;

[0011] Step 1.2: Use the Canny edge detection algorithm in the edge detection algorithm to perform noise reduction on the input original distorted image, then calculate the gradient value and gradient direction, retain the edge pixels, and then use different upper and lower thresholds to detect the edge, and obtain the output as a binary edge image containing edge points;

[0012] Step 1.3: After step 1.2, a binary edge image containing edge points is obtained, and the binary edge image is cut into non-overlapping 32×32 image blocks;

[0013] Step 1.4: Obtain non-overlapping image blocks through step 1.3. The specific method is as follows:

[0014] First, calculate the initial weight of each image block, as shown in formula (1):

[0015] ω=s k / s (1)

[0016] Among them, s k represents the edge points contained in the kth image block in the image block obtained in step 1.3, and s represents the sum of edge points of the binary edge image;

[0017] Then, the initial weights of the obtained image blocks are sorted to obtain M sorted image blocks, and then the image blocks corresponding to the first N weight values ​​are selected as the input of the next stage.

[0018] Step 2 is as follows:

[0019] Step 2.1: Compare the image block numbers obtained in step 1.4 with the image block numbers of the original distorted image obtained in step 1.1, select the image blocks of the original distorted image with the same numbers, and then perform contrast normalization on these image blocks. The calculation formulas are shown in (2) to (4):

[0020]

[0021]

[0022]

[0023] Where (i, j) represents the position of the pixel in the image block of the original distorted image with the same serial number, I(i, j) represents the intensity value of the pixel at the i-th row and j-th column, μ(i, j) represents the mean of the pixel at position (i, j) within the normalized neighborhood calculation range, σ(i, j) represents the variance of the pixel at position (i, j) within the normalized neighborhood calculation range, P and Q represent the neighborhood calculation range of the pixel to be normalized, Represents the normalized pixel value after processing. The final normalized pixel value Composed of new and clearer image blocks;

[0024] A 50-layer deep residual ResNet50 network structure is used to extract features and predict image block scores. The 50-layer deep residual ResNet50 network structure includes four convolutional layers, one pooling layer, and one fully connected layer. Assume that the input of the residual block is x, the convolution layer performs a convolution operation, relu is an activation function, and the output result of the x input after passing through two convolutional layers is F(x). After the skip connection is introduced into the residual network, the output is F(x)+x;

[0025] Step 2.2: The new image block obtained in step 2.1 is used as the input of the deep residual network. The convolution layer is used to extract the features of the input image block. The pooling layer is used to reduce the dimension of the features to reduce the amount of calculation and the number of parameters. Finally, the fully connected layer is used to integrate the features and output the quality score and original weight of the original distorted image block. The original weight is then processed. The calculation formula is shown in (5):

[0026]

[0027] Among them, ε is a constant to ensure that the weight is greater than 0, α q is the original weight of the original distorted image patch, It represents the image block weight of the original distorted image after removing the negative weights.

[0028] In step 2.1, P=Q=3.

[0029] Step 3 is as follows:

[0030] Step 3.1: Obtain the quality score and weight of the image block of the original distorted image through step 2.2, and perform weighted processing, as shown in formula (6):

[0031]

[0032] Among them, y d represents the predicted score of the image block of the dth original distorted image, N represents the number of image blocks obtained after step 2.1, represents the image block weight of the original distorted image obtained after processing, Represents the predicted score value of the weighted original distorted image;

[0033] Step 3.2: Evaluate the quality of the input original image by obtaining the quality prediction score of the original distorted image. A larger score indicates better image quality and a clearer image. A lower score indicates more severe image distortion and a more blurred image.

[0034] The beneficial effect of the present invention is that the edge detection method proposed in this patent is combined with the saliency characteristics of the human eye to improve the accuracy of predicting the score of the original distorted image. At the same time, the use of a deep residual network can better solve the problem of large generalization error in image quality assessment and improve prediction accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is the overall flow chart of the present invention;

[0036] Figure 2 It is a schematic diagram of the residual module in the present invention. DETAILED DESCRIPTION

[0037] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0038] The image quality assessment method of the present invention is based on edge detection and deep residual network, and the flow chart is as follows Figure 1 As shown, the specific implementation steps are as follows:

[0039] Step 1: Input the original distorted image, use the Canny edge detection algorithm to perform edge detection, and divide it into image blocks of the same size, and screen these image blocks;

[0040] Step 1 is as follows:

[0041] Step 1.1: First, crop the original distorted image to obtain image blocks of 32×32 pixels of the same size, label these image blocks, and output the image blocks with serial numbers;

[0042] Step 1.2, edge detection is an important image feature in image processing. The purpose of edge detection is to find a set of pixels in the image with drastic changes in brightness, which are often manifested as contours. In the real world, an edge is often manifested in the image due to discontinuity in depth, discontinuity in surface direction, discontinuity in object material and discontinuity in lighting in the scene. Here we use the Canny edge detection algorithm, which is a relatively outstanding edge detection algorithm, to perform denoising on the input original distorted image, and then calculate the gradient value and gradient direction, retain the edge pixels, and then use different upper and lower thresholds to detect the edge, and the output is a binary edge image containing edge points;

[0043] Step 1.3: After step 1.2, a binary edge image containing edge points is obtained, and the binary edge image is cut into non-overlapping 32×32 image blocks;

[0044] Step 1.4: Obtain non-overlapping image blocks through step 1.3. Since small blocks with more edge points in edge images have stronger significance when observed by human eyes, those image blocks with fewer edge points are discarded. The specific method is as follows:

[0045] First, calculate the initial weight of each image block, as shown in formula (1):

[0046] ω=s k / s (1)

[0047] Among them, s k represents the edge points contained in the kth image block in the image block obtained in step 1.3, and s represents the sum of edge points of the binary edge image;

[0048] Then, the initial weights of the obtained image blocks are sorted to obtain M sorted image blocks, and then the image blocks corresponding to the first N weight values ​​are selected as the input of the next stage.

[0049] Step 2: Normalize the filtered image blocks, input them into the residual network, and output the image block scores and weights of the original distorted image;

[0050] Step 2 is as follows:

[0051] Step 2.1: Compare the image block numbers obtained in step 1.4 with the image block numbers of the original distorted image obtained in step 1.1, select the image blocks of the original distorted image with the same numbers, and then perform contrast normalization on these image blocks. The calculation formulas are shown in (2) to (4):

[0052]

[0053]

[0054]

[0055] Where (i, j) represents the position of the pixel in the image block of the original distorted image with the same serial number, I(i, j) represents the intensity value of the pixel at the i-th row and j-th column, μ(i, j) represents the mean of the pixel at position (i, j) within the normalized neighborhood calculation range, σ(i, j) represents the variance of the pixel at position (i, j) within the normalized neighborhood calculation range, P and Q represent the neighborhood calculation range of the pixel to be normalized, Represents the normalized pixel value after processing. The final normalized pixel value Composed of new and clearer image blocks;

[0056] A smaller normalization window size can improve performance, while a larger normalization window will lead to worse performance. In the no-reference image quality assessment task, there are many types of distortions, and the degree of image distortion varies. Therefore, unlike the image classification task that focuses on the deep features of the image, the effect of using a simple stacked network with increased layers (such as VGG, etc.) is not very good. As the number of layers increases, the shallow image structure information will disappear. Therefore, the 50-layer deep residual ResNet50 network structure is used here to extract features and predict image block scores. The 50-layer deep residual ResNet50 network structure consists of four convolutional layers, one pooling layer, and one fully connected layer, as shown in Table 1 below. The principle of the residual module is as follows: Figure 2 As shown;

[0057] Assume that the input of the residual block is x, the convolution layer performs convolution operation, relu is the activation function, and the output result of x after passing through two convolution layers is F(x). After the skip connection is introduced into the residual network, the output is F(x)+x. The residual module designed in this way can preserve the shallow feature x, thus avoiding the phenomenon of too large generalization error caused by increasing network depth.

[0058] Table 1 Network structure of 50-layer deep residual ResNet50

[0059]

[0060] Step 2.2: The new image block obtained in step 2.1 is used as the input of the deep residual network. The convolution layer is used to extract the features of the input image block. The pooling layer is used to reduce the dimension of the features to reduce the amount of calculation and the number of parameters. Finally, the fully connected layer is used to integrate the features and output the quality score and original weight of the original distorted image block. The original weight is then processed. The calculation formula is shown in (5):

[0061]

[0062] Among them, ε is a constant to ensure that the weight is greater than 0, α q is the original weight of the original distorted image patch, It represents the image block weight of the original distorted image after removing the negative weights.

[0063] In step 2.1, P=Q=3.

[0064] Step 3: Perform weighted processing on the image block scores of the output original distorted image to obtain a predicted value of the image score.

[0065] Step 3 is as follows:

[0066] Step 3.1: Obtain the quality score and weight of the image block of the original distorted image through step 2.2, and perform weighted processing, as shown in formula (6):

[0067]

[0068] Among them, y d represents the predicted score of the image block of the dth original distorted image, N represents the number of image blocks obtained after step 2.1, represents the image block weight of the original distorted image obtained after processing, Represents the predicted score value of the weighted original distorted image;

[0069] Step 3.2: Evaluate the quality of the input original image by obtaining the quality prediction score of the original distorted image. A larger score indicates better image quality and a clearer image. A lower score indicates more severe image distortion and a more blurred image. In practical applications, such as monitoring and fixed-point photography, select images with higher image quality scores.

Claims

1. Image quality assessment method based on edge detection and deep residual network, characterized in that: Follow the steps below to implement it: Step 1: Input the original distorted image, use the Canny edge detection algorithm to perform edge detection, and divide it into image blocks of the same size, and screen these image blocks; Step 1.1: First, crop the original distorted image to obtain image blocks of 32×32 pixels of the same size, label these image blocks, and output the image blocks with serial numbers; Step 1.2: Use the Canny edge detection algorithm in the edge detection algorithm to perform noise reduction on the input original distorted image, then calculate the gradient value and gradient direction, retain the edge pixels, and then use different upper and lower thresholds to detect the edge, and obtain the output as a binary edge image containing edge points; Step 1.3: After step 1.2, a binary edge image containing edge points is obtained, and the binary edge image is cut into non-overlapping 32×32 image blocks; Step 1.4: Obtain non-overlapping image blocks through step 1.3, then sort the initial weights of the obtained image blocks to obtain M sorted image blocks, and then select the image blocks corresponding to the first N weight values ​​as the input of the next stage; Step 2: Normalize the filtered image blocks, input them into the residual network, and output the image block scores and weights of the original distorted image; Step 2.1, comparing the image block serial numbers obtained in step 1.4 with the image block serial numbers of the original distorted image obtained in step 1.1, selecting image blocks of the original distorted image with the same serial numbers, and then performing contrast normalization processing on these image blocks; A 50-layer deep residual ResNet50 network structure is used to extract features and predict image block scores. The 50-layer deep residual ResNet50 network structure includes four convolutional layers, one pooling layer, and one fully connected layer. Assume that the input of the residual block is x, the convolution layer performs a convolution operation, relu is an activation function, and the output result of the x input after passing through two convolutional layers is F(x). After the skip connection is introduced into the residual network, the output is F(x)+x; Step 2.2: The new image block obtained in step 2.1 is used as the input of the deep residual network. The input image block is subjected to feature extraction through the convolution layer, and feature dimension reduction is performed through the pooling layer to reduce the amount of calculation and the number of parameters. Finally, the features are integrated through the fully connected layer, and the quality score and original weight of the original distorted image block are output, and the original weight is processed. Step 3: Perform weighted processing on the image block scores of the output original distorted image to obtain a predicted value of the image score.

2. The image quality assessment method based on edge detection and deep residual network according to claim 1, characterized in that: In step 1.4, non-overlapping image blocks are obtained through step 1.3, and the specific method is as follows: First, calculate the initial weight of each image block, as shown in formula (1): ω=s k / s (1) Among them, s k represents the edge points contained in the kth image block in the image block obtained in step 1.3, and s represents the sum of edge points of the binary edge image.

3. The image quality assessment method based on edge detection and deep residual network according to claim 2 is characterized in that: The specific calculation formula of step 2.1 is shown in (2) to (4): Where (i, j) represents the position of the pixel in the image block of the original distorted image with the same serial number, I(i, j) represents the intensity value of the pixel at the i-th row and j-th column, μ(i, j) represents the mean of the pixel at position (i, j) within the normalized neighborhood calculation range, σ(i, j) represents the variance of the pixel at position (i, j) within the normalized neighborhood calculation range, P and Q represent the neighborhood calculation range of the pixel to be normalized, Represents the normalized pixel value after processing. The final normalized pixel value Composed of new and clearer image blocks; In step 2.2, the original weight is processed, and the specific calculation formula is shown in (5): Among them, ε is a constant to ensure that the weight is greater than 0, α q is the original weight of the original distorted image patch, It represents the image block weight of the original distorted image after removing the negative weights.

4. The image quality assessment method based on edge detection and deep residual network according to claim 3 is characterized in that: In step 2.1, P=Q=3.

5. The image quality assessment method based on edge detection and deep residual network according to claim 3 is characterized in that: The step 3 is as follows: Step 3.1: The quality scores and weights of the image blocks of the original distorted image are obtained through step 2.2, and weighted processing is performed, as shown in formula (6): Among them, y d represents the predicted score of the image block of the dth original distorted image, N represents the number of image blocks obtained after step 2.1, represents the image block weight of the original distorted image obtained after processing, Represents the predicted score value of the weighted original distorted image; Step 3.2: Evaluate the quality of the input original image by obtaining the quality prediction score of the original distorted image. A larger score indicates better image quality and a clearer image. A lower score indicates more severe image distortion and a more blurred image.

Images