Method for processing blurred images based on convolutional neural network

Abstract

The embodiment of the present invention discloses a method for processing blurred images based on a convolutional neural network, which relates to the technical field of computer digital image processing and can realize reference-free image quality evaluation of blurred images. The present invention includes: establishing a supervised image data set, using the supervised image data set to obtain SSIM, and using SSIM as a label of the supervised image data set; establishing a network framework based on CNN; obtaining a loss function corresponding to the network framework, initializing an optimizer and initializing network parameters; inputting a blurred image set into the network framework, and performing forward calculation to obtain a network output, and determining a network loss using the data set label and the network output; performing back propagation calculation using the determined network loss to obtain the gradient values ​​of each network parameter, and updating the network parameters through the optimizer; until the calculated network loss converges, the blurred image quality can be evaluated through the trained network.

Description

Technical Field

[0001] The present invention relates to the technical field of computer digital image processing, and in particular to a method for processing blurred images based on a convolutional neural network. Background Art

[0002] In the process of image and video acquisition, compression, transmission and storage, image distortion and degradation are inevitable. Image blur is one of the main reasons for image quality degradation, which will affect the details and texture of the image and cause a decrease in the visual experience of the image. Therefore, it is necessary to design a corresponding quality evaluation method for blurred images and evaluate the blur degree of the acquired images in real time, so as to provide a data basis for further image improvement and denoising measures.

[0003] Traditional blurred image quality assessment methods generally compare all pixels of a blurred image and its corresponding clear image, and then obtain a score through certain algorithm calculations, such as mean square error (MSE), peak signal to noise ratio (PSNR), structural similarity, etc. However, such methods require obtaining a blurred image and a corresponding clear image in order to score the blurriness of the image. In the process of daily image acquisition and processing, only one blurred image can be obtained, and it is difficult to obtain a corresponding clear reference image. Therefore, such methods are often limited in the evaluation of blurred image quality, and it is difficult to achieve reference-free image quality evaluation of blurred images. Summary of the invention

[0004] An embodiment of the present invention provides a method for processing a blurred image based on a convolutional neural network, which can achieve reference-free image quality evaluation of the blurred image.

[0005] To achieve the above object, the embodiments of the present invention adopt the following technical solutions:

[0006] Step 1, establishing a supervised image dataset, using the supervised image dataset to obtain a structural similarity index (SSIM), and using the SSIM as a label of the supervised image dataset, wherein the supervised image dataset includes a clear image set S1 and a fuzzy image set S2;

[0007] Step 2, establishing a network framework based on a convolutional neural network (CNN), wherein the network framework includes: an image feature extraction network and an image quality estimation network;

[0008] Step 3, obtaining the loss function corresponding to the network framework, initializing the optimizer and initializing the network parameters;

[0009] Step 4, inputting the blurred image set S2 into the network framework, performing forward calculation to obtain the network output, and then determining the network loss using the dataset label and the network output;

[0010] Step 5, using the determined network loss to perform back propagation calculation to obtain the gradient values ​​of each network parameter, and updating the network parameters through the optimizer;

[0011] Step 6, repeating steps 4 to 5 until the network loss calculated in step 4 converges, and obtaining the trained network framework, through which the blurred image can be evaluated for quality.

[0012] The method for processing blurred images based on a convolutional neural network provided in an embodiment of the present invention constructs a supervised image data set, uses a blurred image set and a corresponding clear image set to calculate an SSIM evaluation index as a data set label, and then trains the convolutional neural network constructed by the present invention with the blurred image set and the SSIM label. Finally, the trained network can obtain the corresponding SSIM evaluation index by inputting a blurred image, thereby realizing a reference-free image quality evaluation of the blurred image. The network constructed by the present invention can be trained using a supervised blurred image set and a clear image set, so that the network has the ability to evaluate the quality of blurred images. Compared with traditional image evaluation methods, it is more flexible in use and has better practical value. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0014] Figure 1 A flow chart of a method provided by an embodiment of the present invention;

[0015] Figure 2 A structural diagram of a convolutional neural network provided by an embodiment of the present invention;

[0016] Figure 3 The SSIM estimation curve diagram of the test data set provided by the embodiment of the present invention;

[0017] Figure 4 This is a diagram showing the effect of fuzzy image quality evaluation provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0018] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The embodiments of the present invention will be described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention. It can be understood by those skilled in the art that, unless specifically stated, the singular forms "one", "one", "said" and "the" used herein may also include plural forms. It should be further understood that the term "including" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or their groups. It should be understood that when we say that an element is "connected" or "coupled" to another element, it can be directly connected or coupled to other elements, or there may also be intermediate elements. In addition, the "connection" or "coupling" used here may include wireless connection or coupling. The term "and / or" used herein includes any unit and all combinations of one or more associated listed items. It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as generally understood by those of ordinary skill in the art to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as herein.

[0019] The embodiment of the present invention provides a method for processing blurred images based on a convolutional neural network. Figure 1 As shown, including:

[0020] Step 1: Establish a supervised image dataset, use the supervised image dataset to obtain a structural similarity indicator (SSIM), and use SSIM as a label of the supervised image dataset.

[0021] The supervised image data set includes a clear image set S1 and a blurred image set S2. The clear data set is used to calculate the label data of SSIM with the blurred data set. In this embodiment, the SSIM algorithm requires blurred images and clear images for reference to be calculated. In the subsequent training and testing process, only the label data of SSIM is used without using the clear data set.

[0022] Step 2: Establish a network framework based on a convolutional neural network (CNN), wherein the network framework includes: an image feature extraction network and an image quality estimation network.

[0023] Among them, based on the Convolutional Neural Network (CNN) framework, multiple convolutional layers and pooling layers are used to extract the features of the input image, and multiple fully connected layers are used to estimate and predict the data set labels.

[0024] Step 3, obtain the loss function corresponding to the network framework, initialize the optimizer and initialize the network parameters.

[0025] Step 4: Input the blurred image set S2 into the network framework, perform forward calculation to obtain the network output, and then determine the network loss using the data set label and the network output.

[0026] Among them, the fuzzy image set of the supervised image data set is used as input, and the network output is obtained by forward calculation through the network framework, and the network loss can be calculated in combination with the data set label.

[0027] Step 5, using the determined network loss to perform back propagation calculation to obtain the gradient values ​​of each network parameter, and updating the network parameters through the optimizer.

[0028] Step 6, repeating steps 4 to 5 until the network loss calculated in step 4 converges, and taking the network parameters obtained by the last update as the local optimal solution, to obtain the trained network framework, through which the blurred image quality can be evaluated, for example, when a blurred image is input, an estimated value of its structural similarity index is output to achieve quality evaluation of the blurred image.

[0029] In this embodiment, in step 1, it specifically includes: establishing a clear image set S1 and a blurred image set S2 for image data matching. The blurred image in S2 is obtained by blurring the clear image in S1, the content of the image in S1 corresponds to the content of the image in S2, and an image in S1 and a corresponding image in S2 form an image data pair.

[0030] Further, the supervised image data set is used to obtain SSIM, including: using the mean, standard deviation, and correlation coefficient of the image pixel values ​​to obtain the attribute information of each image data pair in S1 and S2, wherein both S1 and S2 need to be calculated, and the way of obtaining the mean, standard deviation, and correlation coefficient of the image pixel values ​​in S1 and S2 is the same, including:

[0031]

[0032]

[0033]

[0034] Among them, x (i) is the i-th clear image in S1, y (i) is the i-th blurred image of S2, m, n are parameters representing the pixel position of the image, M, N are the length and width of the image respectively, i is a positive integer, represents the mean pixel value of the i-th image in S1, represents the mean pixel value of the i-th image in S2, represents the standard deviation of the pixel values ​​of the i-th image in S1, represents the standard deviation of the pixel values ​​of the i-th image in S2, represents the correlation coefficient of the i-th pair of image pixel values ​​in S1 and S2;

[0035] Afterwards, the attribute information of the obtained image data pair can be used to obtain SSIM. In this embodiment, the general model of SSIM used can be: Where C1=(K1L) 2 , C2=(K2L) 2 , K1=0.01, K2=0.03, L is a dynamic value, for example: L=2 bits per pixel -1.

[0036] Specifically, in this embodiment, the structural similarity index SSIM is used to measure the similarity between two images. Compared with the mean square error MSE and peak signal-to-noise ratio PSNR, it is more in line with the intuitive perception of the human eye, so SSIM is selected as the data set label. SSIM is calculated based on three key features of the image: brightness, contrast, and structure, where brightness is the mean of all pixel values ​​in the image. The attribute information of the image data pair includes brightness, contrast, and structure, where:

[0037] Brightness contrast functions include: are the brightness of the i-th image in S1 and S2 respectively;

[0038] Contrast comparison functions include: are the contrast (standard deviation) of the i-th image in S1 and S2 respectively;

[0039] Structural comparison functions include: in is the correlation coefficient between the i-th image in S1 and the i-th image in S2;

[0040] SSIM is expressed as: Where C1=(K1L0 2 , C2=(K2L) 2 , K1=0.01, K2=0.03, L is a dynamic value, for example: L=2 bits per pixel -1.

[0041] Optionally, in step 1, if the number of images in S1 and S2 is lower than a preset value, S1 and S2 are expanded by rotation and cropping. In the process of constructing a supervised image dataset, for input images of different sizes, the blurred images are resized to the same size to facilitate subsequent network construction and calculation. In addition, in view of the insufficient image data in the dataset, the present invention uses rotation and cropping to perform data enhancement to expand S1 and S2. For example: for each picture in the dataset, the clear image in S1 and the corresponding blurred image in S2 are rotated by a certain angle and different areas of the picture are cropped to obtain a new "blurred image-clear image" pair, and are added to S1 and S2 respectively to achieve data enhancement of the S1 and S2 datasets.

[0042] In this embodiment, it is necessary to establish a network framework based on a convolutional neural network (CNN), which includes: an image feature extraction network and an image quality estimation network, which can be specifically constructed according to the network framework of VGG-16. Specifically in step 2, it includes: performing transfer learning on the image feature extraction network, the image feature extraction network adopts the pre-trained feature extraction network part of VGG16, and the feature data of the image can be extracted through the image feature extraction network. Among them: 5 VGG convolution modules, the convolution layer includes 13 3*3 convolution kernels, and the step size and padding of the convolution layer are both 1; the size of the maximum pooling layer is 5 2*2 pooling units, the step size of the maximum pooling layer is 2 and there is no padding. For example: In order to make the network have better feature extraction capabilities and less network training cost, the network performs transfer learning by using the feature extraction part structure of the VGG-16 network pre-trained on ImageNet and its trained network parameters as the feature extraction network model, which includes 5 VGG convolution modules, namely 13 convolution layers with 3*3 convolution kernels and 5 maximum pooling layers with a pooling unit size of 2*2, where the step size and padding of the convolution layer are 1, the step size of the maximum pooling layer is 2, and there is no padding. Through this feature extraction network part, the input image size can be reduced to 32 times of the original, and the number of channels is increased from 3 to 512; in the embodiment, after the blurred image input after image preprocessing Resize passes through this part, the output feature map size is (512,7,7).

[0043] Furthermore, in step 2, it also includes: in the image quality estimation network, flattening the features output by the image feature extraction network through a Flatten layer, and then predicting the image features to SSIM evaluation indicators through four fully connected modules.

[0044] The four fully connected modules are connected in series, and the activation function of the first three fully connected modules is the Tanh function. The first three fully connected modules also have a BatchNorm layer for normalizing features and a Dropout layer for network regularization. The last fully connected module does not have a BatchNorm layer and a Dropout layer, and the activation function is the Sigmid function. For example: Figure 2 As shown in the figure, the fully connected modules are connected in series, and the final image estimation is the fully connected module part. The four fully connected modules are used to calculate the image features obtained by the feature extraction network to obtain the estimated SSIM. In practical applications, in the fully connected image quality estimation network part, the features extracted from the image are first flattened by the Flatten layer, and then the prediction of the features to the SSIM evaluation index is realized through four fully connected modules. Different from the original VGG-16 network, the activation function of the fully connected module is replaced by the Tanh function from ReLU to avoid the inactivation of a large number of neurons caused by too many negative features, and the BatchNorm layer and Dropout layer are added in the middle of the fully connected module to normalize the features and regularize the network and improve the generalization performance, making the training and prediction of the network more stable. For the last fully connected module that outputs the SSIM estimate, since there is only one output neuron and the SSIM value range is between (0,1), the module does not add the BatchNorm layer and the Dropout layer, and the final activation function is set to the Sigmid function. Specifically, the number of neurons in each fully connected module is FC-3072, FC-3072, FC-128, and FC-1, respectively, and the probability of discarding neurons in the Dropout layer is 0.5.

[0045] In this embodiment, in step 3, it includes: using weighted mean square error (Mean Square Error, MSE) to obtain the loss function.

[0046] Among them, the weight of MSE obeys the SSIM index distribution corresponding to the fuzzy image set S2; the Adam optimization algorithm is used to initialize the optimizer, and the Xavier initialization method is used to randomly initialize the network parameters. Adam is an optimization algorithm used to update the parameters in the network during training. The Xavier initialization method of normal distribution is selected for network parameter initialization. In the preferred embodiment, the momentum term of the optimizer is set to 0.5, the decay rate is 0.999, and the initial learning rate is 0.001.

[0047] The weighted MSE is used to obtain the loss function, including: its regularization weight is calculated and obtained through the fuzzy image set S2 and the SSIM label of the fuzzy image set S2, the SSIM label value of the data set is divided into several intervals, and the occurrence frequency f of the SSIM index of each interval is calculated, so as to further obtain the loss function Among them, the calculation expression of SSIM occurrence frequency f is: n k is the number of image samples with SSIM labels in the interval, N S is the total number of image samples in the S2 blurred image set. Specifically, the loss function is calculated by weighted MSE loss, and its specific calculation method is as follows: Traditional regression problems often assume that their errors obey normal distribution, and use MSE as the loss function to calculate network loss, that is, the SSIM evaluation index is usually concentrated on the side close to 1, and the smaller the SSIM, the lower its probability of occurrence, so the sample imbalance phenomenon of the data set will be more serious. In the process of training the network, the prediction accuracy of images with a large degree of blur is often low. For this reason, the MSE loss is weighted based on the distribution characteristics of the sample labels, so that the network's loss of image samples with a large degree of blur increases. The occurrence frequency f of the SSIM index in each interval is calculated by inputting the blurred image set and its SSIM label. The MSE loss is the mean square error between the network output and the data set label, and the weighted MSE loss function is expressed as: loss = MSE / f. For example: by obtaining the SSIM label distribution of the data set, the loss function weight 1 / f is calculated. If the SSIM size of the ith image is 0.81, and the frequency of 0.8 to 0.,81 in the data set is 0.1, then f = 0.1, and loss = MSE / f.

[0048] In this embodiment, a convolutional neural network is trained using supervised fuzzy image and clear image data sets. The trained network can output the SSIM evaluation index by inputting the fuzzy image as the data, thus realizing the fuzzy image quality evaluation without reference. Compared with the traditional quality evaluation method that needs to refer to the corresponding clear image, the present invention is more flexible and convenient in practical application. The experimental results of the present invention in the embodiment are shown in FIG. Figure 3 and Figure 4 As shown, Figure 3 The SSIM reference value and estimated value curve of the test set data; Figure 4 The SSIM evaluation index for images with different blur levels and their estimates. For example, in the actual application of this embodiment, the maximum number of cycles can be set to 40. When the network training loss does not decrease after 5 consecutive cycles or reaches the maximum number of cycles, the network loss is determined to be converged and the network training is stopped. At this time, the local optimal solution of each parameter of the network is obtained. Through the fully trained network, when a blurred image is input, its structural similarity index estimation value is output to achieve the quality evaluation of the blurred image.

[0049] The method for processing blurred images based on a convolutional neural network provided in an embodiment of the present invention constructs a supervised image data set, uses a blurred image set and a corresponding clear image set to calculate an SSIM evaluation index as a data set label, and then trains the convolutional neural network constructed by the present invention with the blurred image set and the SSIM label. Finally, the trained network can obtain the corresponding SSIM evaluation index by inputting a blurred image, thereby realizing a reference-free image quality evaluation of the blurred image. The network constructed by the present invention can be trained using a supervised blurred image set and a clear image set, so that the network has the ability to evaluate the quality of blurred images. Compared with traditional image evaluation methods, it is more flexible in use and has better practical value.

[0050] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The above is only a specific implementation method of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or replacements that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for processing blurred images based on a convolutional neural network, characterized in that: include: Step 1, establishing a supervised image dataset, using the supervised image dataset to obtain a structural similarity index (SSIM), and using the SSIM as a label of the supervised image dataset, wherein the supervised image dataset includes a clear image set S1 and a fuzzy image set S2; Step 2, establishing a network framework based on a convolutional neural network (CNN), wherein the network framework includes: an image feature extraction network and an image quality estimation network; Step 3, obtaining the loss function corresponding to the network framework, initializing the optimizer and initializing the network parameters; Step 4, inputting the blurred image set S2 into the network framework, performing forward calculation to obtain the network output, and then determining the network loss using the dataset label and the network output; Step 5, using the determined network loss to perform back propagation calculation to obtain the gradient values ​​of each network parameter, and updating the network parameters through the optimizer; Step 6, repeating steps 4 to 5 until the network loss calculated in step 4 converges, obtaining the trained network framework, and performing quality evaluation on the blurred image through the trained network; In step 3, include: The loss function is obtained by using weighted mean square error (MSE), wherein the weight of MSE obeys the SSIM index distribution corresponding to the fuzzy image set S2; the optimizer is initialized by using the Adam optimization algorithm, and the network parameters are randomly initialized by using the Xavier initialization method; The adopting weighted MSE to obtain the loss function includes: The regularization weight is calculated by the fuzzy image set S2 and the SSIM label of the fuzzy image set S2. The SSIM label value of the data set is divided into several intervals, and the occurrence frequency f of the SSIM index in each interval is calculated to further obtain the loss function Among them, the calculation expression of SSIM occurrence frequency f is: n k is the number of image samples with SSIM labels in the interval, N S is the total number of image samples in the S2 blurred image set.

2. The method according to claim 1, characterized in that In step 1, include: A clear image set S1 and a blurred image set S2 are established for image data matching, wherein the blurred image in S2 is obtained by blurring the clear image in S1, the content of the image in S1 corresponds to the content of the image in S2 one by one, and an image in S1 and a corresponding image in S2 constitute an image data pair.

3. The method according to claim 1 or 2, characterized in that: In step 1, obtaining SSIM using the supervised image dataset includes: Obtain the mean, standard deviation and correlation coefficient of the image pixel values ​​in S1 and S2 respectively, and calculate the attribute information of each image data pair in S1 and S2 respectively; The method for obtaining the mean, standard deviation and correlation coefficient of the image pixel values ​​in S1 and S2 is the same, including: Among them, x (i) is the i-th clear image in S1, y (i) is the i-th blurred image of S2, m, n are parameters representing the pixel position of the image, M, N are the length and width of the image respectively, i is a positive integer, represents the mean pixel value of the i-th image in S1, represents the mean pixel value of the i-th image in S2, represents the standard deviation of the pixel values ​​of the i-th image in S1, represents the standard deviation of the pixel values ​​of the i-th image in S2, represents the correlation coefficient of the i-th pair of image pixel values ​​in S1 and S2; The SSIM is obtained by using the attribute information of the obtained image data pair.

4. The method according to claim 3, characterized in that The attribute information of the image data pair includes brightness, contrast and structure, wherein: Brightness contrast functions include: are the brightness of the i-th image in S1 and S2 respectively; Contrast comparison functions include: are the contrast of the i-th image in S1 and S2 respectively; Structural comparison functions include: in is the correlation coefficient between the i-th image in S1 and the i-th image in S2; SSIM is expressed as: Where C1 = (K1L) 2 , C2=(K2L) 2 , K1=0.01, K2=0.03, L is a dynamic value, L=2 bits per pixel -1.

5. The method according to claim 1 or 2, characterized in that: In step 1, include: If the number of images in S1 and S2 is lower than a preset value, S1 and S2 are expanded by rotation and cropping.

6. The method according to claim 1 or 2, characterized in that: In step 2, include: Transfer learning of image feature extraction network, including: 5 VGG convolution modules, the convolution layer includes 13 3*3 convolution kernels, and the stride and padding of the convolution layer are both 1; The size of the maximum pooling layer is 5 2*2 pooling units, the stride of the maximum pooling layer is 2 and there is no padding.

7. The method according to claim 1 or 2, characterized in that: In step 2, it also includes: In the image quality estimation network, the features output by the image feature extraction network are flattened by the Flatten layer, and then the image features are predicted to the SSIM evaluation index by four fully connected modules; The four fully connected modules are connected in series, and the activation function of the first three fully connected modules is the Tanh function. A BatchNorm layer for normalizing features and a Dropout layer for network regularization are also added to the first three fully connected modules. The last fully connected module does not add the BatchNorm layer and the Dropout layer, and the activation function is the Sigmid function.

Images