Image deblurring method and device based on difference between clear image and blurred image

Abstract

The present application discloses an image deblurring method, device and electronic device based on the difference between a clear image and a blurred image. The method comprises: obtaining a clear sample image and a blurred sample image, and determining a difference image between the clear sample image and the blurred sample image; establishing an initial decoding feature model to obtain an initial decoding feature; establishing an initial image deblurring model to obtain a fully trained image deblurring model; obtaining a blurred image, and determining a clear image based on the fully trained image deblurring model. By using the initial decoding feature as supervision, the supervision condition of the initial image deblurring model is increased, and the initial image deblurring model is adjusted inversely according to the feedback of the initial decoding feature, which can accelerate the convergence speed; in addition, based on the differential image adjustment, the difference is magnified, and the deblurring level of the image deblurring model is effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of image deblurring, and in particular to an image deblurring method, device and electronic equipment based on the difference between a clear image and a blurred image. Background Art

[0002] With the development of science and technology, there are more and more electronic products, and images are appearing more and more. However, when the shooting device or object moves, blur artifacts still exist. Blurred images not only cause visual discomfort, but also seriously reduce the performance of the visual system, such as reducing the identification ability of monitoring systems and autonomous driving systems.

[0003] In order to deblur an image, currently, a clear image is mainly obtained through an image restoration network model and other methods to improve the restoration performance. Generally, the final predicted image is supervised by the clear image, but the features of the model decoder module cannot be supervised.

[0004] Therefore, the prior art cannot adjust the model parameters according to the supervised decoding feature method, and there are problems such as slow convergence speed and low deblurring level. Summary of the invention

[0005] In view of this, it is necessary to provide an image deblurring method, device and electronic device based on the difference between a clear image and a blurred image, and adjust the parameters of the image deblurring model by means of supervised decoding features, so as to solve the problems of slow convergence speed and low deblurring level in the prior art.

[0006] In order to solve the above problems, the present invention provides an image deblurring method based on the difference between a clear image and a blurred image, comprising:

[0007] Obtaining a clear sample image and a blurred sample image, and determining a difference image between the clear sample image and the blurred sample image;

[0008] Establishing an initial decoding feature model, inputting the difference image into the initial decoding feature model, taking the first predicted image as output and the difference image as supervision, training the initial decoding feature model, and obtaining initial decoding features;

[0009] Establish an initial image deblurring model, input the blurred sample image into the initial image deblurring model, take the second predicted image and the decoded features as outputs respectively, take the clear sample image and the initial decoded features as supervision, and obtain a fully trained image deblurring model;

[0010] Get a blurry image and determine the sharp image based on a well-trained image deblurring model.

[0011] Furthermore, the blurred sample image is input into the initial image deblurring model, and the second predicted image and the decoded features are respectively output, and the clear sample image and the initial decoded features are used as supervision, so as to obtain a fully trained image deblurring model, including:

[0012] Input the blurred sample image into the initial image deblurring model, and output a second predicted image and a decoded feature;

[0013] According to the second predicted image and the clear sample image, an image loss is obtained by calculating the image loss function;

[0014] According to the decoding features and the initial decoding features, the decoding feature loss is calculated by the decoding feature loss function to obtain the decoding feature loss;

[0015] Determine the loss function of the initial image deblurring model according to the image loss and the decoding feature loss;

[0016] According to the loss function, the parameters of the image deblurring model are adjusted to obtain a fully trained image deblurring model.

[0017] Furthermore, the image loss function includes a mean absolute error loss function and a frequency domain reconstruction loss function.

[0018] Furthermore, according to the image loss and the decoding feature loss, the loss function of the image deblurring model is determined, including:

[0019] According to the image loss and the decoding feature loss, the weight of the image loss and the weight of the decoding feature loss are set, and after a weighted sum operation, the loss function of the image deblurring model is determined.

[0020] Furthermore, according to the loss function, the parameters of the image deblurring model are adjusted to obtain a fully trained image deblurring model, including:

[0021] According to the loss function, the parameters of the image deblurring model are adjusted through the optimizer and the learning rate descent strategy to obtain a fully trained image deblurring model.

[0022] Further, determining a difference image between a clear sample image and a blurred sample image includes:

[0023] The clear sample image and the blurred sample image are subtracted pixel by pixel to obtain a difference image.

[0024] Further, obtaining a clear sample image and a blurred sample image, and determining a difference image between the clear sample image and the blurred sample image, further includes:

[0025] The clear sample image and the blurred sample image are randomly flipped horizontally to obtain a flipped clear sample image and a flipped blurred sample image;

[0026] According to the flipped clear sample image and the flipped blurred sample image, a difference image between the flipped clear sample image and the flipped blurred sample image is obtained by performing pixel-by-pixel subtraction.

[0027] In order to solve the above problem, the present invention also provides an image deblurring device based on the difference between a clear image and a blurred image, comprising:

[0028] A sample acquisition module, used to acquire a clear sample image and a blurred sample image, and determine a difference image between the clear sample image and the blurred sample image;

[0029] An initial decoding feature acquisition module is used to establish an initial decoding feature model, input the difference image into the initial decoding feature model, take the first predicted image as output, take the difference image as supervision, train the initial decoding feature model, and obtain the initial decoding feature;

[0030] An image deblurring model acquisition module is used to establish an initial image deblurring model, input the blurred sample image into the initial image deblurring model, use the second predicted image and the decoded features as outputs, use the clear sample image and the initial decoded features as supervision, and obtain a fully trained image deblurring model;

[0031] The clear image acquisition module is used to acquire the blurred image and determine the clear image based on the well-trained image deblurring model.

[0032] In order to solve the above problems, the present invention also provides an electronic device, including a processor and a memory, wherein a computer program is stored in the memory. When the computer program is executed by the processor, the image deblurring method based on the difference between a clear image and a blurred image as described above is implemented.

[0033] In order to solve the above problems, the present invention also provides a computer-readable storage medium, which stores computer program instructions. When the computer program instructions are executed by a computer, the computer executes the image deblurring method based on the difference between a clear image and a blurred image as described above.

[0034] The beneficial effects of adopting the above technical solution are as follows: the present invention provides an image deblurring method, device and electronic device based on the difference between a clear image and a blurred image, the method comprising: obtaining a clear sample image and a blurred sample image, and determining a difference image between the clear sample image and the blurred sample image; establishing an initial decoding feature model, inputting the difference image into the initial decoding feature model, taking the first predicted image as the output, taking the difference image as the supervision, training the initial decoding feature model, and obtaining an initial decoding feature; establishing an initial image deblurring model, inputting the blurred sample image into the initial image deblurring model, taking the second predicted image and the decoding feature as the output, respectively, taking the clear sample image and the initial decoding feature as the supervision, and obtaining a fully trained image deblurring model; obtaining a blurred image, and determining a clear image based on the fully trained image deblurring model. By taking the initial decoding feature as the supervision, the supervision condition of the initial image deblurring model is increased, and according to the feedback of the initial decoding feature, the initial image deblurring model is adjusted inversely, which can accelerate the convergence speed; in addition, based on the adjustment of the difference image, the difference is magnified, and the deblurring level of the image deblurring model is effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A schematic flow chart of an embodiment of an image deblurring method based on the difference between a clear image and a blurred image provided by the present invention;

[0036] Figure 2 It is a structural diagram of the initial decoding feature model;

[0037] Figure 3 It is a schematic diagram of the structure of the initial image deblurring model;

[0038] Figure 4 A schematic diagram of a flow chart of an embodiment of obtaining a fully trained image deblurring model provided by the present invention;

[0039] Figure 5 A schematic diagram of the structure of an image deblurring device based on the difference between a clear image and a blurred image provided by the present invention;

[0040] Figure 6 This is a structural block diagram of an embodiment of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0041] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0042] Before describing the embodiments, the neural network model and the model are described first:

[0043] A neural network is a model that can be composed of neural units, including input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. Among them, a neural network with many hidden layers is called a deep neural network (DNN). The work of each layer in a neural network can be described by mathematical expressions. From a physical perspective, the work of each layer in a neural network can be understood as completing the transformation from input space to output space (i.e., the row space of a matrix to the column space) through five operations on the input space (a set of input vectors). These five operations include: 1. Dimension increase / reduction; 2. Enlargement / reduction; 3. Rotation; 4. Translation; 5. "Bending". Among them, operations 1, 2, and 3 are completed by the convolution layer, operation 4 is completed by "+b", and operation 5 is implemented by "a()". The word "space" is used here because the classified object is not a single thing, but a class of things. Space refers to the collection of all individuals of this type of thing. Among them, W is the weight matrix of each layer of the neural network, and each value in the matrix represents the weight value of a neuron in this layer. The matrix W determines the spatial transformation from the input space to the output space mentioned above, that is, the W of each layer of the neural network controls how to transform the space. The purpose of training a neural network is to eventually obtain the weight matrices of all layers of the trained neural network. Therefore, the training process of a neural network is essentially learning how to control spatial transformation, more specifically learning the weight matrix.

[0044] In the embodiments of the present application, the model used to process the image enhancement task is essentially a neural network or a part of the model structure is a neural network. The application of the model generally includes two stages: training and reasoning. The training stage is used to train the model according to the training set to obtain the trained model; the reasoning stage is used to use the trained model to perform image enhancement processing on real unlabeled instances (i.e., the real target image to be processed), and the quality of the enhanced image obtained after the image enhancement processing is one of the important indicators to measure the quality of a model training.

[0045] In particular, the network model in this application refers to a model including an encoder-decoder structure of upsampling and downsampling.

[0046] At present, the parameters of the deblurred image model are generally adjusted by supervising the final predicted image through a clear image, that is, the model parameters are adjusted according to the obtained results, rather than according to the decoded features of the model. Therefore, there is a problem that the model parameters cannot be adjusted by supervising the decoded features.

[0047] In order to solve the problem in the prior art that model parameters cannot be adjusted according to supervised decoding features, the present invention provides an image deblurring method, device and electronic device based on the difference between a clear image and a blurred image, which are described in detail below.

[0048] like Figure 1 As shown, Figure 1 A flow chart of an embodiment of an image deblurring method based on the difference between a clear image and a blurred image provided by the present invention includes:

[0049] Step S101: obtaining a clear sample image and a blurred sample image, and determining a difference image between the clear sample image and the blurred sample image.

[0050] Step S102: Establish an initial decoding feature model, input the difference image into the initial decoding feature model, use the first predicted image as output, use the difference image as supervision, train the initial decoding feature model, and obtain initial decoding features.

[0051] Step S103: Establish an initial image deblurring model, input the blurred sample image into the initial image deblurring model, use the second predicted image and the decoded features as outputs respectively, use the clear sample image and the initial decoded features as supervision, and obtain a fully trained image deblurring model.

[0052] Step S104: Obtain a blurred image, and determine a clear image based on a well-trained image deblurring model.

[0053] In this embodiment, first, a clear sample image and a blurred sample image are obtained, and secondly, a difference image between the clear sample image and the blurred sample image is obtained based on the clear sample image and the blurred sample image; then, an initial decoding feature model is established, and an initial decoding feature is obtained based on the difference image; next, an initial image deblurring model is established, and the blurred sample image is input into the initial image deblurring model, and a fully trained image deblurring model is obtained with the second predicted image and the decoding feature as outputs, respectively, and the clear sample image and the initial decoding feature as supervision; finally, a blurred image is obtained, and a clear image is determined based on the fully trained image deblurring model.

[0054] It can be understood that in the above embodiment, the initial decoding features are first obtained based on the difference image, and then the initial decoding features are used as supervision to adjust the initial image deblurring model to obtain a fully trained image deblurring model. By using the initial decoding features as supervision, not only the supervision of the initial image deblurring model is increased, but also the initial image deblurring model is adjusted inversely according to the feedback of the initial decoding features, which can accelerate the convergence speed; in addition, based on the difference image adjustment, the difference is magnified, and the deblurring level of the image deblurring model is effectively improved.

[0055] As a preferred embodiment, in step S101, in order to obtain a difference image between a clear sample image and a blurred sample image, the clear sample image and the blurred sample image are subtracted pixel by pixel to obtain a difference image.

[0056] Furthermore, in order to obtain more difference images between clear sample images and blurred sample images based on limited clear sample images and blurred sample images, the clear sample images and blurred sample images can be randomly flipped horizontally to obtain flipped clear sample images and flipped blurred sample images respectively; then, the difference image between the flipped clear sample image and the flipped blurred sample image is obtained by performing pixel-by-pixel difference.

[0057] As a preferred embodiment, in step S102, in order to obtain the initial decoding features, an initial decoding feature model is established, the difference image is input into the initial decoding feature model, and the first predicted image is output. The initial decoding feature model is trained with the difference image as supervision. After the trained decoding feature model is obtained, the difference image is input into the trained decoding feature model again to obtain the initial decoding features.

[0058] Furthermore, in order to increase the number of difference images between the clear sample image and the blurred sample image and enhance the training effect of the initial decoding feature model, the difference image between the clear sample image and the blurred sample image may be enhanced.

[0059] In a specific embodiment, in order to quantitatively determine whether the initial decoding feature model is fully trained, the training effect of the initial decoding feature model is represented by a loss function, wherein the formula of the final loss function of the initial decoding feature model is:

[0060] L cont1 =‖D-P1‖1

[0061]

[0062] L1=L cont +λL FR

[0063] Among them, L cont1 is the mean absolute error loss function of the decoding feature model, L FR1 is the frequency domain reconstruction loss function of the decoding feature model, P1 is the predicted image output by the initial decoding feature model, D is the pixel-by-pixel difference image between the clear image and the blurred image, is the two-dimensional fast Fourier transform, λ is the weight of the frequency domain reconstruction loss function, and L1 is the final loss function of the initial decoding feature model.

[0064] In this embodiment, by calculating the final loss function of the initial decoding feature model, the training result of the initial decoding feature model can be quantitatively determined, thereby improving the training effect of the initial decoding feature model.

[0065] Furthermore, in order to better adjust the parameters in the initial decoding feature model, a suitable optimizer and learning rate reduction strategy can be selected to optimize the initial decoding feature model, thereby improving the training effect.

[0066] It should be noted that the initial decoding feature model and the initial image deblurring model in the above embodiments are both network models established based on the encoding-decoding architecture.

[0067] As a preferred embodiment, in order to facilitate comparison and obtain better initial decoding features, the encoder and decoder of the initial decoding feature model and the initial image deblurring model are respectively the same.

[0068] like Figure 2 As shown, Figure 2 Schematic diagram of the structure of the initial decoding feature model, including: encoder 201, decoder 202, convolution layer 203. The encoder 201 encodes the initial image to obtain an encoded image, and then the decoder 202 decodes the encoded image to obtain a decoding feature, and finally, the convolution layer 203 is used to obtain a predicted image corresponding to the initial image.

[0069] The initial decoding feature model takes the difference image as input and the first predicted image as output, and uses the difference image as supervision to adjust the parameters of the initial decoding feature model, thereby obtaining the initial decoding features.

[0070] In a specific embodiment, the encoder 201 and the decoder 202 both include original-size features and two-level downsampled features, and each time the number of channels of the downsampled feature map is doubled, while the length and width become half of the original.

[0071] like Figure 3 As shown, Figure 3 The structure diagram of the initial image deblurring model includes: an encoder 301, a decoder 302, a first convolutional layer 303 and a second convolutional layer 304. The encoder 301 encodes the initial image to obtain an encoded image, and then the decoder 302 decodes the encoded image, and then passes through the first convolutional layer 303 to obtain a predicted image corresponding to the initial image, and finally passes through the second convolutional layer 304 to obtain a decoding feature.

[0072] Among them, the initial image deblurring model takes the blurred image as input, the clear sample image and the initial decoding features as supervision, the predicted image as the first output, and the decoding features as the second output. By comparing the decoding features with the initial decoding features, the parameters of the initial image deblurring model are adjusted to obtain a better image deblurring model.

[0073] In a specific embodiment, the encoder 301 and the decoder 302 both include original-size features and two-level downsampled features, and each time the number of channels of the downsampled feature map is doubled, while the length and width become half of the original.

[0074] In a specific embodiment, the original size of the initial image deblurring model has 32 channels, and both the length and width are 256, that is, the third-level feature has 128 channels, and both the length and width are 64. The convolution kernel is 3×3, which is used to convert the decoded feature with 128 channels into a 3-channel predicted image.

[0075] In other embodiments, the initial decoding features may be obtained in other ways, for example, the corresponding initial decoding features may be obtained according to historical records, or a database may be formed after long-term accumulation, and then the corresponding initial decoding features may be obtained in the database.

[0076] In the above embodiment, the encoder 301 is completely identical to the encoder 201, and the decoder 302 is completely identical to the decoder 202, so as to ensure that the initial image deblurring model can obtain better results with the initial decoding features as supervision during the training process; when the encoder and decoder in the initial image deblurring model are not completely identical to the encoder and decoder in the initial decoding feature model, the initial decoding features as supervision will lose their reference significance and cannot achieve the goal of improving the training effect of the initial image deblurring model.

[0077] As a preferred embodiment, in step S103, in order to obtain a fully trained image deblurring model, first, an initial image deblurring model is established; secondly, the blurred sample image is input into the initial image deblurring model, with the predicted image and the decoded features as outputs, respectively, and the clear sample image and the initial decoded features as supervision; finally, a fully trained image deblurring model is obtained.

[0078] Preferably, in order to obtain a well-trained image deblurring model, such as Figure 4 As shown, Figure 4 A schematic diagram of a process for obtaining a fully trained image deblurring model according to an embodiment of the present invention includes:

[0079] Step S131: input the blurred sample image into the initial image deblurring model, and output a second predicted image and decoded features.

[0080] Step S132: according to the second predicted image and the clear sample image, the image loss is calculated by using the image loss function.

[0081] Step S133: According to the decoding features and the initial decoding features, the decoding feature loss is calculated by a decoding feature loss function to obtain the decoding feature loss.

[0082] Step S134: Determine the loss function of the initial image deblurring model according to the image loss and the decoding feature loss.

[0083] Step S135: According to the loss function, adjust the parameters of the image deblurring model to obtain a fully trained image deblurring model.

[0084] In this embodiment, with clear sample images and initial decoded features as supervision, the loss function of the initial image deblurring model is determined by image loss and decoded feature loss, and then the parameters of the initial image deblurring model are adjusted according to the loss function, and training is iteratively performed to obtain a fully trained image deblurring model.

[0085] It can be understood that in this embodiment, clear sample images and initial decoding features are used as supervision to implement a dual-condition, feedback-adjusted initial image deblurring model, and the initial decoding features obtained based on the decoding feature model are used as supervision to improve the training effect of the image deblurring model, thereby effectively improving the accuracy of the fresh image obtained according to the image deblurring model.

[0086] Preferably, in order to improve the training effect of the initial image deblurring model, in this embodiment, a quantitative evaluation is performed by calculating the final loss function of the initial image deblurring model.

[0087] In a specific embodiment, a blurred image is used as input, and the mean absolute error loss function and the frequency domain reconstruction loss function are calculated for the clear image and the second predicted image, and a certain ratio is used as the sum, that is, the loss supervised by the clear image is calculated; at the same time, the initial decoding features and the decoding features are used to calculate the mean absolute error loss function, that is, the loss supervised by the initial decoding features is calculated; finally, according to the weight of the image loss and the weight of the decoding feature loss, a weighted sum operation is performed to obtain the final loss function of the initial image deblurring model.

[0088] Among them, the formula of the loss function includes:

[0089] L cont2 =‖S-P2‖1

[0090]

[0091] L feature =‖F1-F2‖1

[0092] L2=L cont2 +λ1L FR2 +λ2L feature

[0093] Among them, L cont2 is the mean absolute error loss function of the image deblurring model, L FR2 is the frequency domain reconstruction loss function of the image deblurring model, L feature is the initial decoding feature and the loss function of the decoding feature, P2 is the predicted image output by the initial image deblurring model, S is the clear image, is the two-dimensional fast Fourier transform, F1 is the initial decoding feature, F2 is the decoding feature, and L2 is the final loss function of the initial image deblurring model.

[0094] Furthermore, after obtaining the final loss function of the initial image deblurring model, a suitable optimizer and learning rate descent strategy can be selected to adjust the parameters of the image deblurring model in a targeted manner, thereby enhancing the training effect.

[0095] Finally, after the fully trained image deblurring model is obtained, in step S104, any blurred image can be obtained by only needing the fully trained image deblurring model to determine the clear image corresponding to the blurred image.

[0096] Furthermore, in order to clearly illustrate the process of obtaining a clear image based on a blurred image by an image deblurring model, this application uses the GoPro dataset to illustrate the training process by way of example.

[0097] Among them, the GoPro dataset consists of 2103 pairs of training sets and 1111 pairs of test sets with a resolution of 1280×720.

[0098] Furthermore, in order to produce a more realistic blurred image, the GoPro dataset does not use the method of convolving the clear image with a blur kernel, but uses a high-speed camera to quickly record a series of clear images, and then averages these clear images with very short intervals to obtain a blurred image. Because the camera sensor continuously receives light during the exposure process, we can understand that the sensor continuously receives signals from the clear image during exposure, which together form a blurred image.

[0099] The synthetic image is converted into discrete pixel values ​​through a nonlinear CRF (Camera Response Function) to generate a digital blurred image. Therefore, a high-speed camera can be used to simulate this process.

[0100] The process of fuzzy accumulation and generation can be expressed as:

[0101]

[0102] Where T is the exposure time, S(t is the sensor signal (the clear image obtained at time t); M is the number of frames of the clear image obtained by sampling, and S[i] is the i-th blurred image and the clear sampled image. The blurred image and the clear image are CRF transformations.

[0103] Since the clear signal is accumulated and averaged before the CRF transformation, and in practice, the original sensor signal and CRF are unknown, in order to restore the imaging process as much as possible, the above uses the approximation of CRF, the gamma curve with γ = 2.2:

[0104]

[0105] Furthermore, in order to reduce the memory usage during model training, 256×256 blocks of the same position are randomly cropped from the clear-blurred image pair, and the two blocks are randomly flipped horizontally at the same time with a flipping probability of 0.5. The flipped (or unflipped) blocks are then input into the decoding feature model as pixel-by-pixel difference. The input has two levels of output after passing through the encoding and decoding module, but only its predicted output is needed in the training phase. The mean absolute error loss function and frequency domain reconstruction loss function are calculated for the input and the predicted image of the decoding feature model, and the final loss function is obtained by calculating the ratio of 1:0.1.

[0106] Among them, the formula for calculating the final loss function of the decoding feature model is:

[0107] L cont =‖D-P1‖1

[0108]

[0109] L1=L cont +0.1×L FR

[0110] Among them, P1 is the predicted image output by the decoding feature model, and D is the pixel-by-pixel difference image between the clear image and the blurred image. is the two-dimensional fast Fourier transform.

[0111] Furthermore, in order to adjust the parameters of the initial decoding feature model according to the final loss of the decoding feature model, the Adam optimizer is selected, and the learning rate reduction strategy selects the cosine annealing strategy from 2×10 -4 Down to 1×10 -6 , train for 500 rounds and obtain a fully trained decoding feature model.

[0112] After the decoding feature model is trained, the pre-trained weights are saved as a .pkl file and imported into the decoding feature model. If the difference image is input again to the decoding feature model with the pre-trained weights, the initial decoding features of the decoding feature model can be obtained.

[0113] When training the image deblurring model, similarly, randomly crop a 256×256 block at the same position from the clear-blurred image pair, and randomly flip both blocks horizontally at the same time with a flip probability of 0.5. The cropped block of the blurred image is used as input, and the mean absolute error loss function and the frequency domain reconstruction loss function are calculated for the cropped block of the clear image and the predicted image of the image deblurring model, and the sum is taken at a ratio of 1:0.1 as the clear image supervision loss. At the same time, the loss with the initial decoded features as supervision is calculated, and the clear image supervision loss and the decoding feature supervision loss are added at a ratio of 1:0.01 as the final loss function of the image deblurring model.

[0114] Among them, the formula of the loss function includes:

[0115] L cont =‖S-P2‖1

[0116]

[0117] L feature =‖F1-F2‖1

[0118] L2=L cont +0.1×L FR +0.01L feature

[0119] Among them, P2 is the predicted image output by the image deblurring model, S is the clear image, is a two-dimensional fast Fourier transform, F1 is the decoded feature output by the decoding feature model, F2 is the decoded feature output by the image deblurring model, and L2 is the final loss function of the training image deblurring model.

[0120] Similarly, we choose to use the Adam optimizer and follow the learning rate reduction strategy of the cosine annealing strategy, starting from 2×10 -4 Down to 1×10 -6 , and train for 1000 rounds. After the training is completed, a fully trained image deblurring model is obtained.

[0121] Through the above method, the initial decoding features are obtained according to the initial decoding feature model, and then the initial image deblurring model is supervised and trained according to the initial decoding features, thereby increasing the supervision of the initial image deblurring model. According to the feedback of the initial decoding features, not only the initial image deblurring model is reversely adjusted to accelerate the convergence speed; but also based on the difference image adjustment, the difference is amplified, thereby effectively improving the deblurring level of the image deblurring model.

[0122] In order to test the effect of this technical solution, a comparative experiment was set up.

[0123] In the comparative experiment, a model identical to the image deblurring model was established, called the comparative model. Initial decoding feature supervision training was not used during training, and the rest of the configuration and training methods were exactly the same as those used to train the image deblurring model. The comparative model and the image deblurring model were tested for their effectiveness. Blurred images of the test set in the GoPro dataset were input into the two models respectively, and the PSNR of the clear images whose outputs corresponded to the blurred images was calculated.

[0124] Finally, the PSNR (peak signal-to-noise ratio) of the image deblurring model on the Gopro test set is 31.06dB, while the comparison model is only 30.94dB.

[0125] In other words, this technical solution can effectively improve the deblurring level of the image.

[0126] In order to solve the above problems, the present invention also provides an image deblurring device based on the difference between a clear image and a blurred image. Figure 5 As shown, Figure 5 The structure diagram of the image deblurring device based on the difference between a clear image and a blurred image provided by the present invention is as follows. The image deblurring device 500 based on the difference between a clear image and a blurred image includes:

[0127] The sample acquisition module 501 is used to acquire a clear sample image, a blurred sample image, and a difference image between the clear sample image and the blurred sample image;

[0128] An initial decoding feature acquisition module 502 is used to obtain an initial decoding feature according to a difference image between a clear sample image and a blurred sample image;

[0129] The image deblurring model acquisition module 503 is used to establish an initial image deblurring model, input the blurred sample image into the initial image deblurring model, take the predicted image and the decoded features as outputs, take the clear sample image and the initial decoded features as supervision, and obtain a fully trained image deblurring model;

[0130] The clear image acquisition module 504 is used to acquire the blurred image and determine the clear image based on the well-trained image deblurring model.

[0131] The present invention also provides an electronic device, such as Figure 6 As shown, Figure 6 This is a structural block diagram of an embodiment of an electronic device provided by the present invention. The electronic device 600 may be a computing device such as a mobile terminal, a desktop computer, a notebook, a PDA, and a server. The electronic device 600 includes a processor 601 and a memory 602, wherein the memory 602 stores an image deblurring program 603 based on the difference between a clear image and a blurred image.

[0132] In some embodiments, the memory 602 may be an internal storage unit of a computer device, such as a hard disk or memory of the computer device. In other embodiments, the memory 602 may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the computer device. Further, the memory 602 may also include both an internal storage unit of the computer device and an external storage device. The memory 602 is used to store application software and various types of data installed in the computer device, such as program codes for installing the computer device. The memory 602 may also be used to temporarily store data that has been output or is to be output. In one embodiment, an image deblurring program 603 based on the difference between a clear image and a blurred image may be executed by the processor 601, thereby realizing the image deblurring method based on the difference between a clear image and a blurred image in each embodiment of the present invention.

[0133] In some embodiments, the processor 601 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run program codes or process data stored in the memory 602, such as executing an image deblurring program based on the difference between a clear image and a blurred image.

[0134] This embodiment also provides a computer-readable storage medium, on which is stored an image deblurring program based on the difference between a clear image and a blurred image. When the computer program is executed by a processor, the image deblurring method based on the difference between a clear image and a blurred image as described in any of the above technical solutions is implemented.

[0135] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0136] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. An image deblurring method based on the difference between a clear image and a blurred image, characterized in that: include: Acquire a clear sample image and a blurred sample image, and determine a difference image between the clear sample image and the blurred sample image; Establishing an initial decoding feature model, inputting the difference image into the initial decoding feature model, taking the first predicted image as output and the difference image as supervision, training the initial decoding feature model to obtain initial decoding features; Establish an initial image deblurring model, input the blurred sample image into the initial image deblurring model, and output a second predicted image and a decoded feature; calculate the image loss through an image loss function according to the second predicted image and the clear sample image; calculate the decoding feature loss through a decoding feature loss function according to the decoding feature and the initial decoding feature; determine the loss function of the initial image deblurring model according to the image loss and the decoding feature loss; adjust the parameters of the image deblurring model according to the loss function to obtain a fully trained image deblurring model; Obtaining a blurred image, and determining a clear image based on the trained image deblurring model; Wherein, the loss function of the decoding feature model includes a mean absolute error loss function and a frequency domain reconstruction loss function; The formula for the final loss function of the initial decoding feature model is: in, is the mean absolute error loss function, is the frequency domain reconstruction loss function, is the first predicted image, A pixel-by-pixel difference image is obtained between the clear image and the blurred image. is the two-dimensional fast Fourier transform, is the weight of the frequency domain reconstruction loss function, is the final loss function of the initial decoding feature model.

2. The image deblurring method based on the difference between a clear image and a blurred image according to claim 1, characterized in that: The step of determining the loss function of the initial image deblurring model according to the image loss and the decoding feature loss comprises: According to the image loss and the decoding feature loss, the weight of the image loss and the weight of the decoding feature loss are set, and after a weighted sum operation, the loss function of the image deblurring model is obtained.

3. The image deblurring method based on the difference between a clear image and a blurred image according to claim 1, characterized in that: The step of adjusting the parameters of the image deblurring model according to the loss function to obtain a fully trained image deblurring model comprises: According to the loss function, the parameters of the image deblurring model are adjusted through an optimizer and a learning rate descent strategy to obtain a fully trained image deblurring model.

4. The image deblurring method based on the difference between a clear image and a blurred image according to claim 1, characterized in that: The determining of the difference image between the clear sample image and the blurred sample image includes: The clear sample image and the blurred sample image are subtracted pixel by pixel to obtain the difference image.

5. The image deblurring method based on the difference between a clear image and a blurred image according to claim 1, characterized in that: The step of acquiring a clear sample image and a blurred sample image, and determining a difference image between the clear sample image and the blurred sample image, further includes: Randomly flipping the clear sample image and the blurred sample image horizontally to obtain a flipped clear sample image and a flipped blurred sample image; According to the flipped clear sample image and the flipped blurred sample image, a difference image between the flipped clear sample image and the flipped blurred sample image is obtained by performing pixel-by-pixel subtraction.

6. An image deblurring device based on the difference between a clear image and a blurred image, characterized in that: include: A sample acquisition module, used to acquire a clear sample image and a blurred sample image, and determine a difference image between the clear sample image and the blurred sample image; An initial decoding feature acquisition module is used to establish an initial decoding feature model, input the difference image into the initial decoding feature model, take the first predicted image as output, take the difference image as supervision, train the initial decoding feature model, and obtain initial decoding features; An image deblurring model acquisition module, used for establishing an initial image deblurring model, inputting the blurred sample image into the initial image deblurring model, and outputting a second predicted image and a decoding feature; According to the second predicted image and the clear sample image, an image loss is calculated by an image loss function; according to the decoded features and the initial decoded features, a decoding feature loss is calculated by a decoding feature loss function; according to the image loss and the decoding feature loss, a loss function of the initial image deblurring model is determined; according to the loss function, parameters of the image deblurring model are adjusted to obtain a fully trained image deblurring model; A clear image acquisition module, used to acquire a blurred image and determine a clear image based on the trained image deblurring model; Wherein, the loss function of the decoding feature model includes a mean absolute error loss function and a frequency domain reconstruction loss function; The formula for the final loss function of the initial decoding feature model is: in, is the mean absolute error loss function, is the frequency domain reconstruction loss function, is the first predicted image, A pixel-by-pixel difference image is obtained between the clear image and the blurred image. is the two-dimensional fast Fourier transform, is the weight of the frequency domain reconstruction loss function, is the final loss function of the initial decoding feature model.

7. An electronic device, characterized in that: The method comprises a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the image deblurring method based on the difference between a clear image and a blurred image as described in any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium, characterized in that: The storage medium stores computer program instructions, and when the computer program instructions are executed by a computer, the computer is caused to execute the image deblurring method based on the difference between a clear image and a blurred image as claimed in any one of claims 1 to 5.

Images