Image deblurring method and device based on convolutional sparse coding feature insertion network

Abstract

The invention particularly relates to an image deblurring method and device based on a convolutional sparse coding feature insertion network, and the method comprises the steps: obtaining a to-be-processed blurred image; and inputting the to-be-processed blurred image to a pre-trained image deblurring model, extracting depth features from the to-be-processed blurred image through the pre-trained image deblurring model, and outputting a clear image of the to-be-processed blurred image according to the depth features of the to-be-processed blurred image, the pre-trained image deblurring model is obtained by training a preset recovery network by using the blurred image training set and a reference image training set corresponding to the blurred image training set based on a preset sparse coding algorithm. Therefore, through the convolution sparse coding feature insertion network, the clear reference database and the blurred image training set are combined, depth feature extraction and reference feature connection are utilized, the clear image is output, the problem that in the prior art, deblurring cannot be carried out on the blurred image of spatial transformation is solved, and the deblurring effect is improved.

Description

Technical Field

[0001] This application relates to the technical fields of digital image processing, pattern recognition, computer vision, and deep learning, and particularly relates to an image deblurring method and device based on a convolutional sparse coding feature insertion network. Background Art

[0002] Image deblurring is an important task in the fields of computer vision and image processing, and its purpose is to restore a clear image signal from a blurred image. In the process of the continuous development of the deep learning network framework, neural networks have also become the main force in the deblurring task. Sparse coding, also known as sparse dictionary learning, is a representation learning method that aims to represent input data as a sparse linear combination of basic elements, which are called "atoms" and form a "dictionary". The atoms in the dictionary are generally not orthogonal to each other, and it can be overcomplete, so that the signal dimension of the dictionary can be higher than the observed signal dimension. Convolutional sparse coding is a sub-paradigm of sparse coding, where the dictionary coding can be regarded as an extremely sparse circulant matrix. For the input observed signal, it can be expressed as a linear combination of a very small number of atoms in the dictionary, or as the product of the dictionary coding and a very sparse matrix.

[0003] In related technologies, image deblurring methods usually utilize the prior information of the image and build a virtual blur kernel for the blur situation of the data set, establish a parameterized blur model for image blur, and thus establish an optimization problem.

[0004] However, there is a large gap between academic research and industrial applications. For example, the image aberration degradation of digital cameras and the signal processing system often face complex mixed degradation problems, while most academic methods are designed based on simulated degradation situations and are difficult to generalize to practical applications. The image deblurring methods in related technologies are difficult to restore the image with high quality for complex factors or blurs in the spatial domain, time domain, or frequency domain transformation, and urgent solutions are needed. Summary of the Invention

[0005] This application provides an image deblurring method and device based on a convolutional sparse coding feature insertion network to solve problems such as the inability of related technologies to deblur images blurred by spatial transformation and improve the deblurring effect.

[0006] The first aspect embodiment of this application provides an image deblurring method based on a convolutional sparse coding feature insertion network, including the following steps:

[0007] Obtain a to-be-processed blurred image;

[0008] Input the to-be-processed blurred image into a pre-trained image deblurring model. The pre-trained image deblurring model extracts depth features from the to-be-processed blurred image, and outputs a clear image of the to-be-processed blurred image according to the depth features of the to-be-processed blurred image. Among them, the pre-trained image deblurring model is obtained by training a preset recovery network based on a preset sparse coding algorithm, using a blurred image training set and a reference image training set corresponding to the blurred image training set.

[0009] Optionally, before inputting the to-be-processed blurred image into the pre-trained image deblurring model, it further includes:

[0010] Obtain a reference database and a blurred data set, where the reference database includes multiple images that meet a preset resolution;

[0011] Perform image processing on the multiple images that meet the preset resolution to obtain the reference image training set, and based on a preset sparse coding algorithm, obtain a sparse linear combination of basic elements according to the reference image training set, and obtain a clear reference feature library according to the sparse linear combination of the basic elements;

[0012] Perform projection processing and upsampling resolution expansion processing on the clear reference feature library to obtain a processed clear reference feature library that meets a preset size; at the same time, perform slicing processing, random inversion transformation processing, and spatial sampling processing on the images in the blurred data set to obtain the blurred image training set;

[0013] Based on the to-be-trained blurred image data and the processed clear reference feature library, train the preset recovery network until the training result meets the preset training conditions to obtain the pre-trained image deblurring model.

[0014] Optionally, based on the to-be-trained blurred image data and the processed clear reference feature library, training the preset recovery network includes:

[0015] Input the to-be-trained blurred image data set into the preset recovery network to extract depth features, and obtain the depth features of each blurred image;

[0016] Connect the processed clear reference feature library with the depth features of each blurred image, and based on the connection result, use a preset recovery loss function to optimize and output a recovered clear image.

[0017] Optionally, the performing image processing on the multiple images that meet the preset resolution to obtain a training data set includes:

[0018] Perform color-to-gray processing, contrast transformation processing, and spatial sampling processing on the multiple images that meet the preset resolution respectively, and obtain the training dataset according to the processing results.

[0019] Optionally, the sparse linear combination of the basic elements is:

[0020]

[0021] where D is the dictionary, z i is the i-th encoding, N is the number of samples, x i is the i-th data point, β is the Lagrange multiplier, d j is the j-th column vector in D, and L is the number of columns of D.

[0022] An embodiment of the second aspect of the present application provides an image deblurring device based on a convolutional sparse coding feature insertion network, including:

[0023] An acquisition module, configured to acquire a to-be-processed blurred image;

[0024] A processing module, configured to input the to-be-processed blurred image into a pre-trained image deblurring model, extract depth features from the to-be-processed blurred image through the pre-trained image deblurring model, and output a clear image of the to-be-processed blurred image according to the depth features of the to-be-processed blurred image, where the pre-trained image deblurring model is obtained by training a preset recovery network based on a preset sparse coding algorithm using a blurred image training set and a reference image training set corresponding to the blurred image training set.

[0025] Optionally, before inputting the to-be-processed blurred image into the pre-trained image deblurring model, the processing module is further configured to:

[0026] Acquire a reference database and a blurred dataset, where the reference database includes multiple images that meet the preset resolution;

[0027] Perform image processing on the multiple images that meet the preset resolution to obtain the reference image training set, and based on a preset sparse coding algorithm, obtain a sparse linear combination of basic elements according to the reference image training set, and obtain a clear reference feature library according to the sparse linear combination of the basic elements;

[0028] Perform projection processing and upsampling resolution expansion processing on the clear reference feature library to obtain a processed clear reference feature library that meets the preset size; at the same time, perform slicing processing, random inversion transformation processing, and spatial sampling processing on the images in the blurred dataset to obtain the blurred image training set;

[0029] Based on the to-be-trained blurred image data and the processed clear reference feature library, train the preset restoration network until the training result meets the preset training conditions, and obtain the pre-trained image de-blurring model.

[0030] Optionally, the processing module is specifically configured to:

[0031] Input the to-be-trained blurred image data set into a preset restoration network to extract depth features, and obtain the depth features of each blurred image;

[0032] Connect the processed clear reference feature library with the depth features of each blurred image, and based on the connection result, use a preset restoration loss function to optimize and output a restored clear image.

[0033] Optionally, the processing module is specifically configured to:

[0034] Perform color-to-gray processing, contrast transformation processing, and spatial sampling processing on the multiple images that meet the preset resolution respectively, and obtain the training data set according to the processing results.

[0035] Optionally, the sparse linear combination of the basic elements is:

[0036]

[0037] where D is the dictionary, z i is the i-th encoding, N is the number of samples, x i is the i-th data point, β is the Lagrange multiplier, d j is the j-th column vector in D, and L is the number of columns of D.

[0038] An embodiment of the third aspect of the present application provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and the instructions are set to execute the image de-blurring method based on the convolutional sparse coding feature insertion network as described in the above embodiments.

[0039] An embodiment of the fourth aspect of the present application provides a computer-readable storage medium, on which a computer program is stored, and the program is executed by a processor to implement the image de-blurring method based on the convolutional sparse coding feature insertion network as described in the above embodiments.

[0040] An embodiment of the fifth aspect of the present application provides a computer program product, the computer program product stores a computer program, and when the program is executed by a processor, it implements the image de-blurring method based on the convolutional sparse coding feature insertion network as described in the above embodiments.

[0041] Accordingly, the embodiment of the present application obtains a to-be-processed blurred image, inputs the to-be-processed blurred image into a pre-trained image deblurring model, extracts deep features from the to-be-processed blurred image through the pre-trained image deblurring model, and outputs a clear image of the to-be-processed blurred image according to the deep features of the to-be-processed blurred image. Among them, the pre-trained image deblurring model is obtained by training a preset recovery network based on a preset sparse coding algorithm using a blurred image training set and a reference image training set corresponding to the blurred image training set. Accordingly, through a convolutional sparse coding feature insertion network, combining a clear reference database and a blurred image training set, using deep feature extraction and reference feature connection, a clear image is output, solving problems such as the inability to deblur a spatially transformed blurred image in the related art and improving the deblurring effect.

[0042] Additional aspects and advantages of the present application will be given in part in the following description, become apparent in part from the following description, or be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The above and / or additional aspects and advantages of the present application will become apparent and be readily understood from the following description of the embodiments in conjunction with the drawings, in which:

[0044] Figure 1 FIG. is a flowchart of an image deblurring method based on a convolutional sparse coding feature insertion network according to an embodiment of the present application;

[0045] Figure 2 FIG. is a schematic diagram of an extraction module responsible for extracting convolutional sparse representations and dictionaries and a training module responsible for training a blurred image to be restored to a clear image in an image deblurring method based on a convolutional sparse coding feature insertion network according to an embodiment of the present application;

[0046] Figure 3 FIG. is a block schematic diagram of an image deblurring device based on a convolutional sparse coding feature insertion network according to an embodiment of the present application;

[0047] Figure 4 FIG. is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0048] The embodiments of the present application will be described in detail below. The examples of the embodiments are shown in the drawings, in which the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions from beginning to end. The embodiments described below with reference to the drawings are exemplary and are intended to explain the present application and should not be construed as limiting the present application.

[0049] The following describes the image deblurring method and device based on a convolutional sparse coding feature insertion network according to an embodiment of the present application. Aiming at the problem that the related technologies in the above-mentioned background art cannot deblur images with spatially-varying blur, the present application provides an image deblurring method based on a convolutional sparse coding feature insertion network. In this method, a to-be-processed blurred image is obtained, and the to-be-processed blurred image is input into a pre-trained image deblurring model. The pre-trained image deblurring model extracts deep features from the to-be-processed blurred image, and outputs a clear image of the to-be-processed blurred image according to the deep features of the to-be-processed blurred image. Among them, the pre-trained image deblurring model is obtained by training a preset recovery network based on a preset sparse coding algorithm using a blurred image training set and a reference image training set corresponding to the blurred image training set. Thus, through the convolutional sparse coding feature insertion network, combined with a clear reference database and a blurred image training set, by using deep feature extraction and reference feature connection, a clear image is output, solving problems such as the inability of related technologies to deblur images with spatially-varying blur, and improving the deblurring effect.

[0050] Specifically, Figure 1 FIG. is a flowchart of an image deblurring method based on a convolutional sparse coding feature insertion network provided by an embodiment of the present application.

[0051] As Figure 1 shown, the image deblurring method based on a convolutional sparse coding feature insertion network includes the following steps:

[0052] In step S101, a to-be-processed blurred image is obtained.

[0053] Among them, the to-be-processed blurred image refers to the input image that needs to be deblurred, and usually the image sharpness decreases due to reasons such as camera shake, object movement, or inaccurate focusing during the shooting process.

[0054] Specifically, the to-be-processed blurred image can be obtained by direct input, that is, the user uploads one or more blurred images (such as mobile phone photos, surveillance video frames, etc.), blurred images that can be captured in real time by imaging devices such as cameras and scanners, and can be loaded from a preset blurred image database (such as synthetic or real datasets of motion blur, defocus blur, etc.) through dataset calls. The to-be-processed blurred image should satisfy high resolution, rich high-frequency features, and diverse semantic information; it is preferred to select images taken by commercial or industrial-grade single-lens reflex cameras, which can minimize noise, distortion, and errors, and at the same time provide a relatively high resolution (it is recommended to exceed 1000 pixels).

[0055] In step S102, the to-be-processed blurred image is input into a pre-trained image deblurring model. The pre-trained image deblurring model extracts depth features from the to-be-processed blurred image and outputs a clear image of the to-be-processed blurred image according to the depth features of the to-be-processed blurred image. Among them, the pre-trained image deblurring model is obtained by training a preset restoration network based on a preset sparse coding algorithm using a blurred image training set and a corresponding reference image training set of the blurred image training set.

[0056] Among them, the depth feature refers to a high-level abstract representation automatically extracted from the input data through a deep neural network. The preset sparse coding algorithm is a representation learning method aimed at decomposing the input data into a sparse linear combination of basic elements.

[0057] Optionally, in some embodiments, before inputting the to-be-processed blurred image into the pre-trained image deblurring model, it further includes: obtaining a reference database and a blurred data set, where the reference database includes multiple images that meet the preset resolution; performing image processing on the multiple images that meet the preset resolution to obtain a reference image training set, and based on the preset sparse coding algorithm, obtaining a sparse linear combination of basic elements according to the reference image training set, and obtaining a clear reference feature library according to the sparse linear combination of basic elements; performing projection processing and upsampling resolution expansion processing on the clear reference feature library to obtain a processed clear reference feature library that meets the preset size; at the same time, performing slicing processing, random inversion transformation processing, and spatial sampling processing on the images in the blurred data set to obtain a blurred image training set; training the preset restoration network based on the to-be-trained blurred image data and the processed clear reference feature library until the training result meets the preset training conditions to obtain the pre-trained image deblurring model.

[0058] Optionally, in some embodiments, the sparse linear combination of basic elements is:

[0059]

[0060] Among them, D is the dictionary, z i is the i-th encoding, N is the number of samples, x i is the i-th data point, β is the Lagrange multiplier, d j is the j-th column vector in D, and L is the number of columns of D.

[0061] Among them, the preset resolution is the image size or resolution preset to ensure the model training effect and processing efficiency. The preset size is the standardized size setting for the images or feature maps input into the neural network to ensure the model training and processing efficiency. The preset training conditions are the goals or standards set when training the image deblurring model. When the model reaches these goals or standards, it is considered that the training is completed.

[0062] It is understandable that the number of images meeting the preset resolution in the database should reach a certain scale (it is recommended to exceed 1000) to enhance the extensiveness of the data and the robustness of model training; the dataset itself needs to be a real dataset and contain image degradation features such as noise and imaging errors so that the network can be successfully trained with complete fitting. Based on the preset sparse coding algorithm, for a given image dataset The goal of the sparse coding algorithm is to represent each data point x i as a linear combination of several atoms from an overcomplete dictionary i.e., a sparse linear combination of basic elements, and the encoding z needs to satisfy a certain sparsity constraint. Assuming x i is noise-free, the above recovery problem can be mathematized as: i

[0063]

[0064] where is the minimum 0-norm encoding that satisfies the data as a linear combination of the dictionary, D is an overcomplete dictionary, x i is the image dataset, and z i is the i-th encoding.

[0065] To extend the optimization scope to the entire reference image training set, the above optimization problem can be reformulated as a sparse linear combination of basic elements, where D = [d1,…,d i ,…,d L . If no constraints are added to D, the joint optimization of the dictionary and the encoding will face great challenges, and optimizing a high-dimensional and overcomplete dictionary representation will consume a large amount of time and computing resources. Common basic methods are Basis Pursuit and Orthogonal Matching Pursuit, which can provide necessary constraint conditions for the uniqueness and stability of the solved dictionary and encoding. Similarly, to improve the efficiency of convolution operations, in the same way as the solution method of sparse coding, the spatial domain convolution is converted into a frequency domain operation, thus significantly improving the computing performance. At the same time, continue to use the reparameterization method to directly optimize the matrix after frequency domain conversion to reduce the numerical solution complexity brought by Fourier transform or inverse transform; the online solution of the optimization problem at the current moment can be reformulated as:

[0066]

[0067] where is to intercept the first M elements of the vector, t is the number of iteration steps of the optimization problem, P is the number of rows of D, ​is the frequency-domain transform of D, and k is the serial number of the column, is the frequency-domain transform of D but without intercepting the edge block part.

[0068] The clear reference feature library is obtained according to the sparse linear combination of basic elements, and the extracted sparse features are mapped to a spatial representation more suitable for deep learning model processing; in order to match the requirements of the model input layer or ensure consistency with the blurred image dataset, it is necessary to adjust the resolution of the images in the clear reference feature library; Upsampling (also known as interpolation) is a method to increase the image size, which can be achieved by different interpolation algorithms, such as nearest neighbor interpolation, bilinear interpolation or bicubic interpolation, etc.; Selecting an appropriate interpolation method can effectively maintain the quality of the image and avoid distortion.

[0069] Furthermore, the images in the dataset are converted into RGB images through image source information and then into grayscale images; after local contrast normalization, edge fading processing is used to randomly blur the edges of the Gaussian; at the connection part between the encoder and the decoder, deep feature processing is performed on the trained dictionary; through multiple upsampling modules, its resolution is adjusted to be consistent with the feature resolution at the bottleneck, and then it is fused with the deep features at the end of the encoder and input into the decoder to obtain the blurred image training set. The blurred image dataset is used as input and fed into a preset restoration network to optimize the network parameters. Repeat the above steps and continuously adjust the network parameters until the training results meet the preset training conditions. The preset training conditions include but are not limited to the loss function value being lower than a certain threshold, the accuracy rate reaching a predetermined standard, and after a certain number of iteration cycles, the performance improvement amplitude is extremely small or no longer improves, etc.

[0070] Optionally, in some embodiments, training a preset restoration network based on the to-be-trained blurred image data and the processed clear reference feature library includes: inputting the to-be-trained blurred image dataset into the preset restoration network to extract deep features, and obtaining the deep features of each blurred image; connecting the processed clear reference feature library with the deep features of each blurred image, and based on the connection result, using the preset restoration loss function to optimize and output the restored clear image.

[0071] Among them, the preset restoration network refers to a pre-designed deep learning model architecture specifically used to restore clear images from blurred images. The preset restoration loss function is a function that measures the difference between the model output result and the real clear image, thereby guiding the parameter optimization in the model training process. Deep features are high-level, multi-scale feature representations automatically learned from input images through deep neural networks.

[0072] It can be understood that the convolutional sparse coding method (CSC) is used to extract deep high-frequency coding features from image pixels. The core objective is to decompose the input data into a sparse linear combination of basic elements, which are called "atoms" and form a "dictionary". The atoms in the dictionary usually do not have orthogonality and may be over-complete, which means that the signal dimension of the dictionary can exceed the dimension of the observed signal. This property enables the same signal vector to be described by multiple different representation methods, thereby enhancing the sparsity and flexibility of the coding. Using an online learning algorithm, the convolutional sparse coding and dictionary of a large-scale database are solved quickly in batches. According to the theory of sparse coding, the optimization of the dictionary and the optimization of the coding need to be carried out step by step. The Alternating Direction Method of Multipliers (ADMM) algorithm is used for solution, and finally the dictionary and sparse coding are generated to form a clear reference feature library.

[0073] It should be noted that as Figure 2 shown, Figure 2It is a schematic diagram of an extraction module responsible for extracting convolutional sparse representation and dictionary and a training module responsible for training the restoration of a blurred image to a clear image in an image deblurring method based on a convolutional sparse coding feature insertion network according to an embodiment of the present application. The distorted image generated by the optical simulation system and its corresponding standard image can be used as the blurred image data to be trained and input into a preset restoration network. The preset restoration network framework refers to the basic design of UNet. The encoding part includes an initial processing module, four downsampling convolutional layers, and a pooling layer. Each convolutional unit consists of two 3x3 filtering operations and a ReLU activation unit. At the end of the module, the feature matrix is compressed by a 2x2 max pooling operation to gradually reduce the spatial dimension of the feature matrix. The number of feature channels output by each filtering layer and downsampling layer is 64, 128, 256, 512, and 1024 respectively. As the number of channels increases, the spatial resolution gradually decreases, thereby enhancing the perception ability of the preset restoration network. The decoding part consists of four upsampling modules. Each module first expands the feature matrix through a 2x2 transposed convolution operation to restore it to the resolution of the corresponding encoding layer. In addition, feature integration is achieved through cross-layer connections between the encoder and the decoder. The feature matrix of the encoder is merged with the feature matrix of the corresponding layer of the decoder to retain both global context information and local details. After merging the feature matrices transmitted through the cross-layer connection, two 3x3 filtering operations and ReLU activation units are applied again to gradually repair the detailed information. At the intersection of the encoder and the decoder, deep feature processing is performed on the trained dictionary. Its resolution is adjusted to be consistent with the bottleneck feature through multiple upsampling modules, and then it is fused with the deep feature at the end of the encoder and input into the decoder. Finally, the mean square error (MSE) is used as the loss function for optimization, and the restored clear image is output.

[0074] Optionally, in some embodiments, image processing is performed on multiple images meeting a preset resolution to obtain a training dataset, including: performing color-to-gray conversion processing, contrast transformation processing, and spatial sampling processing on multiple images meeting the preset resolution respectively, and obtaining the training dataset according to the processing results.

[0075] It can be understood that after preparing an image database with unified dimensions, a specific data normalization method should be adopted to reduce the sensitivity of subsequent modules to the variance of data distribution. Specifically, through techniques such as color-to-gray conversion, contrast adjustment, and spatial sampling, the color gamut, contrast, and spatial dimensions of the images can be unified into a standardized color space. The color-to-gray processing means that for each input color image, it can be converted into a grayscale image by calculating the weighted average of the red, green, and blue channel values of each pixel. The contrast transformation processing can adjust the contrast of the image through techniques such as histogram equalization or adaptive histogram equalization. The spatial sampling processing can be achieved through upsampling and downsampling. Downsampling is achieved by reducing the image size, for example, using methods such as nearest neighbor interpolation, bilinear interpolation, or bicubic interpolation; upsampling is when it is necessary to increase the image resolution, the above interpolation methods can also be used to enlarge the image size.

[0076] According to the image deblurring method based on a convolutional sparse coding feature insertion network proposed by an embodiment of the present application, the embodiment of the present application obtains a to-be-processed blurred image, inputs the to-be-processed blurred image into a pre-trained image deblurring model, extracts deep features from the to-be-processed blurred image through the pre-trained image deblurring model, and outputs a clear image of the to-be-processed blurred image according to the deep features of the to-be-processed blurred image, wherein the pre-trained image deblurring model is obtained by training a preset restoration network based on a preset sparse coding algorithm using a blurred image training set and a reference image training set corresponding to the blurred image training set. Thus, through the convolutional sparse coding feature insertion network, combining a clear reference database and a blurred image training set, using deep feature extraction and reference feature connection, a clear image is output, solving problems such as the inability of related technologies to deblur blurred images with spatial transformation, and improving the deblurring effect.

[0077] Next, an image deblurring device based on a convolutional sparse coding feature insertion network proposed by an embodiment of the present application will be described with reference to the accompanying drawings.

[0078] Figure 3 It is a block diagram of an image deblurring device based on a convolutional sparse coding feature insertion network according to an embodiment of the present application.

[0079] As Figure 3 shown, the image deblurring device 10 based on a convolutional sparse coding feature insertion network includes: an acquisition module 100 and a processing module 200.

[0080] Among them, the acquisition module 100 is used to acquire a to-be-processed blurred image;

[0081] The processing module 200 is configured to input the to-be-processed blurred image into a pre-trained image deblurring model, extract depth features from the to-be-processed blurred image through the pre-trained image deblurring model, and output a clear image of the to-be-processed blurred image according to the depth features of the to-be-processed blurred image. The pre-trained image deblurring model is obtained by training a preset recovery network based on a preset sparse coding algorithm using a blurred image training set and a reference image training set corresponding to the blurred image training set.

[0082] Optionally, before inputting the to-be-processed blurred image into the pre-trained image deblurring model, the processing module 200 is further configured to: obtain a reference database and a blurred data set, where the reference database includes multiple images meeting a preset resolution; perform image processing on the multiple images meeting the preset resolution to obtain a reference image training set, and based on the preset sparse coding algorithm, obtain a sparse linear combination of basic elements according to the reference image training set, and obtain a clear reference feature library according to the sparse linear combination of basic elements; perform projection processing and upsampling resolution expansion processing on the clear reference feature library to obtain a processed clear reference feature library meeting a preset size; at the same time, perform slicing processing, random inversion transformation processing, and spatial sampling processing on the images in the blurred data set to obtain a blurred image training set; train the preset recovery network based on the to-be-trained blurred image data and the processed clear reference feature library until the training result meets a preset training condition to obtain the pre-trained image deblurring model.

[0083] Optionally, the processing module 200 is specifically configured to: input the to-be-trained blurred image data set into the preset recovery network to extract depth features, and obtain the depth features of each blurred image; connect the processed clear reference feature library with the depth features of each blurred image, and based on the connection result, use a preset recovery loss function for optimization to output a recovered clear image.

[0084] Optionally, the processing module 200 is specifically configured to: perform color-to-gray processing, contrast transformation processing, and spatial sampling processing on the multiple images meeting the preset resolution respectively, and obtain a training data set according to the processing results.

[0085] Optionally, the sparse linear combination of basic elements is:

[0086]

[0087] where D is a dictionary, z i is the i-th code, N is the number of samples, x i is the i-th data point, β is a Lagrange multiplier, d j is the j-th column vector in D, and L is the number of columns in D.

[0088] It should be noted that the foregoing explanation of the embodiments of the image deblurring method based on the convolutional sparse coding feature insertion network also applies to the image deblurring method device based on the convolutional sparse coding feature insertion network of this embodiment, and will not be elaborated here.

[0089] According to the image deblurring method device based on the convolutional sparse coding feature insertion network provided by the embodiments of the present application, the embodiments of the present application obtain a to-be-processed blurred image, input the to-be-processed blurred image into a pre-trained image deblurring model, extract deep features from the to-be-processed blurred image through the pre-trained image deblurring model, and output a clear image of the to-be-processed blurred image according to the deep features of the to-be-processed blurred image. Among them, the pre-trained image deblurring model is obtained by training a preset restoration network based on a preset sparse coding algorithm using a blurred image training set and a reference image training set corresponding to the blurred image training set. Thus, through the convolutional sparse coding feature insertion network, combined with the clear reference database and the blurred image training set, using deep feature extraction and reference feature connection, a clear image is output, solving problems such as the inability of related technologies to deblur blurred images with spatial transformation, and improving the deblurring effect.

[0090] Figure 4 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application. The electronic device may include:

[0091] A memory 401, a processor 402, and a computer program stored on the memory 401 and executable on the processor 402.

[0092] When the processor 402 executes the program, it implements the image deblurring method based on the convolutional sparse coding feature insertion network provided in the above embodiments.

[0093] Furthermore, the electronic device further includes:

[0094] A communication interface 403 for communication between the memory 401 and the processor 402.

[0095] The memory 401 is used to store a computer program executable on the processor 402.

[0096] The memory 401 may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory.

[0097] If the memory 401, the processor 402, and the communication interface 403 are implemented independently, the communication interface 403, the memory 401, and the processor 402 can be interconnected through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 only a thick line is used to represent it in Figure 4 , but it does not mean that there is only one bus or one type of bus.

[0098] Optionally, in a specific implementation, if the memory 401, the processor 402, and the communication interface 403 are integrated on a single chip, the memory 401, the processor 402, and the communication interface 403 can communicate with each other through an internal interface.

[0099] The processor 402 may be a Central Processing Unit (CPU), or an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0100] The embodiments of the present application also provide a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the image deblurring method based on the convolutional sparse coding feature insertion network as described above is implemented.

[0101] The embodiments of the present application also provide a computer program product. The computer program product stores a computer program, and when the program is executed by a processor, the image deblurring method based on the convolutional sparse coding feature insertion network as described above is implemented.

[0102] In the description of this specification, the descriptions referring to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of this application. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or N embodiments or examples. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0103] In addition, the terms "first" and "second" are used only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one of such features. In the description of this application, the meaning of "N" is at least two, such as two, three, etc., unless otherwise specifically and clearly defined.

[0104] Any process or method description shown in the flowchart or described in other ways herein can be understood to represent a module, segment, or part of code including one or more N executable instructions for implementing a customized logical function or process, and the scope of the preferred embodiments of this application includes additional implementations, where the functions can be executed in a substantially simultaneous manner or in a reverse order according to the involved functions, rather than in the order shown or discussed, which should be understood by those skilled in the technical field to which the embodiments of this application belong.

[0105] It should be understood that each part of this application can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application specific integrated circuits having appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.

[0106] Those of ordinary skill in the art of this technology can understand that all or part of the steps carried by the method of implementing the above embodiments can be completed by instructing relevant hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

Claims

1. An image deblurring method based on a convolutional sparse coding feature insertion network, characterized in that Including the following steps: Obtain a blurred image to be processed; Input the blurred image to be processed into a pre-trained image deblurring model. Extract depth features from the blurred image to be processed through the pre-trained image deblurring model, and output a clear image of the blurred image to be processed according to the depth features of the blurred image to be processed. Among them, the pre-trained image deblurring model is obtained by training a preset restoration network based on a preset sparse coding algorithm using a blurred image training set and a reference image training set corresponding to the blurred image training set.

2. The method according to claim 1, characterized in that, Before inputting the blurred image to be processed into the pre-trained image deblurring model, it further includes: Obtain a reference database and a blurred data set, where the reference database includes multiple images meeting a preset resolution; Perform image processing on the multiple images meeting the preset resolution to obtain the reference image training set, and based on a preset sparse coding algorithm, obtain a sparse linear combination of basic elements according to the reference image training set, and obtain a clear reference feature library according to the sparse linear combination of the basic elements; Perform projection processing and upsampling resolution expansion processing on the clear reference feature library to obtain a processed clear reference feature library meeting a preset size; at the same time, perform slicing processing, random inversion transformation processing and spatial sampling processing on the images in the blurred data set to obtain the blurred image training set; Train the preset restoration network based on the blurred image data to be trained and the processed clear reference feature library until the training result meets a preset training condition to obtain the pre-trained image deblurring model.

3. The method according to claim 2, wherein Training the preset restoration network based on the blurred image data to be trained and the processed clear reference feature library includes: Input the blurred image data set to be trained into the preset restoration network to extract depth features, and obtain the depth features of each blurred image; Connect the processed clear reference feature library with the depth features of each blurred image, and based on the connection result, use a preset restoration loss function to optimize and output a restored clear image.

4. The method according to claim 2, wherein The performing image processing on the multiple images meeting the preset resolution to obtain a training data set includes: Perform color-to-gray processing, contrast transformation processing and spatial sampling processing on the multiple images meeting the preset resolution respectively, and obtain the training data set according to the processing results.

5. The method according to claim 2, wherein The sparse linear combination of the basic elements is: Among them, D is a dictionary, z i is the i-th encoding, N is the number of samples, x i is the i-th data point, β is the Lagrange multiplier, d j is the j-th column vector in D, and L is the number of columns of D.

6. An image deblurring device based on a convolutional sparse coding feature insertion network, characterized in that Including: An acquisition module, configured to acquire a blurred image to be processed; A processing module, configured to input the blurred image to be processed into a pre-trained image deblurring model. Extract depth features from the blurred image to be processed through the pre-trained image deblurring model, and output a clear image of the blurred image to be processed according to the depth features of the blurred image to be processed. Among them, the pre-trained image deblurring model is obtained by training a preset restoration network based on a preset sparse coding algorithm using a blurred image training set and a reference image training set corresponding to the blurred image training set.

7. The device according to claim 6, characterized in that, Before inputting the to-be-processed blurred image into the pre-trained image deblurring model, the processing module is further configured to: Obtain a reference database and a blurred dataset, wherein the reference database includes multiple images that meet a preset resolution; Perform image processing on the multiple images that meet the preset resolution to obtain a reference image training set, and based on a preset sparse coding algorithm, obtain a sparse linear combination of basic elements according to the reference image training set, and obtain a clear reference feature library according to the sparse linear combination of the basic elements; Perform projection processing and upsampling resolution expansion processing on the clear reference feature library to obtain a processed clear reference feature library that meets a preset size; at the same time, perform slicing processing, random inversion transformation processing and spatial sampling processing on the images in the blurred dataset to obtain a blurred image training set; Train the preset restoration network based on the to-be-trained blurred image data and the processed clear reference feature library until the training result meets a preset training condition, so as to obtain the pre-trained image deblurring model.

8. An electronic device, characterized in that, Including: A memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor executes the program to implement the image deblurring method based on a convolutional sparse coding feature insertion network according to any one of claims 1-5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to be used to implement the image deblurring method based on a convolutional sparse coding feature insertion network according to any one of claims 1-5.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the image deblurring method based on a convolutional sparse coding feature insertion network according to any one of claims 1-5.

Images