Image deblurring method, device and apparatus

Abstract

The present invention discloses an image processing method, apparatus, medium and equipment, which relate to the field of artificial intelligence and are used to remove moving objects in an image. The method specifically includes: extracting original image information of three channels from a blurred image; performing Laplace transform and mean square error calculation on the original image information of the three channels respectively to obtain intermediate image information corresponding to the three channels; averaging the intermediate image information corresponding to the three channels to obtain the blurriness of the blurred image; and selecting a corresponding deep learning network model to perform a deblurring operation on the blurred image according to the blurriness of the blurred image.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image deblurring method, device and equipment. Background Art

[0002] In recent years, dual-lens and even multi-lens smart cameras have become widely used, such as in dual-camera smartphones, autonomous vehicles, and drones. However, out-of-focus blur is unavoidable in many scenarios. For example, device shake, inaccurate focus, or rapid object movement during capture can all cause image blur. This not only affects the quality of the captured image but also has a serious impact on subsequent image processing. Numerous traditional deblurring algorithms exist, which can be categorized into blind deconvolution and non-blind deconvolution, depending on whether or not a priori models are used. The former makes certain prior assumptions about the convolution kernel or the image, while the latter assumes a known blur kernel function. Both traditional algorithms are very difficult to solve, and often struggle to achieve satisfactory results when the blur type is unknown or the blur level is high.

[0003] Currently popular image deblurring algorithms often require a large convolutional neural network. While these networks can handle a variety of distortion types and varying degrees of blur, the high complexity of these networks incurs significant computational overhead. Furthermore, because these algorithms use the same neural network architecture to process all images, some images still struggle to achieve satisfactory results after processing.

[0004] Therefore, how to select an algorithm with lower computational overhead for image deblurring operations has become an urgent problem that needs to be solved in the industry. Summary of the Invention

[0005] Embodiments of the present invention provide an image deblurring method, apparatus, and electronic device for selecting different deep learning network models for deblurring operations according to the degree of image blur, thereby obtaining images of better quality while reducing computational overhead.

[0006] In a first aspect, the present invention provides an image deblurring method comprising the following steps:

[0007] Extracting original image information of three channels from a blurred image; performing Laplace transform and mean square error calculation on the original image information of the three channels respectively to obtain intermediate image information corresponding to the three channels; averaging the intermediate image information corresponding to the three channels to obtain the blurriness of the blurred image; and selecting a corresponding deep learning network model to perform a deblurring operation on the blurred image according to the blurriness of the blurred image.

[0008] The beneficial effects of the image deblurring method provided by the present invention are: on the one hand, by adopting a smaller network model to estimate the blurriness of the blurred image, image preprocessing is avoided, and the original information of the image is retained as much as possible while greatly reducing the computational overhead; on the other hand, different deep learning models are selected according to different blurriness to perform deblurring operations. In this way, some images with lower blurriness can use smaller deep learning models for deblurring than images with higher blurriness, further reducing the computational overhead.

[0009] In a possible embodiment, selecting a corresponding deep learning network model to perform a deblurring operation on the blurred image according to the blurriness of the blurred image includes:

[0010] When the blurriness of the blurred image is greater than or equal to a set threshold, a gradually upsampling deep learning network model is used to deblur the blurred image; when the blurriness of the blurred image is lower than the set threshold, a two-layer parallel deep learning network model is used to deblur the blurred image.

[0011] In this method, since the deblurring operation of images with high blurriness requires more network resources, a two-layer parallel deep learning network model is used for deblurring operations. While meeting the basic requirements of deblurring, it can retain as much color and image detail information as possible.

[0012] In a possible embodiment, the gradually upsampling deep learning network model includes a first feature extraction branch, a second feature extraction branch, a fully connected module, and an image reconstruction branch, each feature extraction branch is composed of a different network model, the first feature extraction branch is used to extract a first feature map from the blurred image; the second feature extraction branch is used to extract a second feature map from the blurred image;

[0013] A fully connected module, configured to fully connect the first feature map and the second feature map to obtain a third feature map;

[0014] The image reconstruction branch is used to add the blurred image to the first feature map to obtain a first intermediate image of the next level; add the intermediate image to the second feature map to obtain a second intermediate image of the next level; and finally add the third feature map obtained after full connection to the second intermediate image to obtain a clear image.

[0015] In a possible embodiment, the two-layer parallel deep learning network model includes a first-layer deep learning network model and a second-layer deep learning network model;

[0016] The first layer of deep learning network model is used to retain the color information and information of the image and obtain the first feature map;

[0017] The first layer of deep learning network model is used for deblurring operation to obtain the second feature map;

[0018] The superposition result of the first feature map and the second feature map is equal to a clear image.

[0019] In a second aspect, an embodiment of the present invention further provides an image deblurring device, comprising modules / units for executing the method of any possible embodiment of the first aspect. These modules / units may be implemented in hardware, or in hardware executing corresponding software implementations.

[0020] In a third aspect, an embodiment of the present invention provides an electronic device comprising a processor and a memory. The memory is configured to store one or more computer programs; when the one or more computer programs stored in the memory are executed by the processor, the server is enabled to implement the method of any possible embodiment of the first aspect described above.

[0021] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium includes a computer program. When the computer program runs on an electronic device, the electronic device executes a method of any possible embodiment of any of the above aspects.

[0022] In a fifth aspect, an embodiment of the present invention further provides a method comprising a computer program product, which, when the computer program product is run on a terminal, enables the electronic device to execute any possible embodiment of any of the above aspects.

[0023] For the beneficial effects of the second to fifth aspects, reference may be made to the descriptions of the first and second aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0025] Figure 1 A schematic diagram of an application scenario provided by an embodiment of the present invention;

[0026] Figure 2 A flowchart of an image deblurring method provided by an embodiment of the present invention;

[0027] Figure 3A schematic flow chart of another image deblurring method provided by an embodiment of the present invention;

[0028] Figure 4A A flowchart of a method for image deblurring based on a gradually upsampling deep learning network model provided by an embodiment of the present invention;

[0029] Figure 4B A schematic diagram illustrating the principle of a gradually upsampling deep learning network model provided by an embodiment of the present invention;

[0030] Figure 5 A schematic flow chart of a method for image deblurring using a dual-layer parallel deep learning network model provided in an embodiment of the present invention;

[0031] Figure 6 A schematic flow chart of another image deblurring method provided by an embodiment of the present invention;

[0032] Figure 7 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0033] In order to improve the shortcomings of existing image deblurring, the present invention provides an image deblurring method, which can select different deep learning network models for deblurring operations based on the degree of image blur, thereby obtaining images of better quality while reducing computational overhead.

[0034] Some terms used in the embodiments of the present invention are explained below to facilitate understanding by those skilled in the art.

[0035] 1. Convolutional Neural Network

[0036] Convolutional neural networks (CNNs) are a type of feedforward neural network with a deep structure that incorporates convolutional computations. They are a representative algorithm for deep learning. CNNs possess the ability to learn representations and perform translation-invariant classification of input information based on their hierarchical structure. Inspired by research in biological neuroscience, CNNs were initially developed to process data with network-like structures, such as images that can be viewed as two-dimensional networks composed of pixels. The general architecture of a CNN consists of a data input layer, a convolutional layer, a data activation layer, a pooling layer, a fully connected layer, and a data output layer.

[0037] The embodiments of the present invention relate to artificial intelligence (AI) and machine learning technology, and are designed based on deep learning networks and machine learning (ML) in artificial intelligence.

[0038] With the research and advancement of artificial intelligence technology, artificial intelligence has been studied and applied in many fields, such as common smart homes, smart customer service, virtual assistants, smart speakers, smart marketing, unmanned driving, autonomous driving, robots, smart medical care, etc. I believe that with the development of technology, artificial intelligence will be applied in more fields and play an increasingly important role.

[0039] Machine learning is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, and inductive learning.

[0040] In the description of the embodiments of the present invention, the terms used in the following embodiments are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, the singular expressions "a", "said", "above", "the" and "this" are intended to also include expressions such as "one or more", unless there is a clear contrary indication in the context. It should also be understood that in the following embodiments of the present invention, "at least one", "one or more" refer to one or more (including two). The term "and / or" is used to describe the association relationship of associated objects, indicating that three relationships can exist; for example, A and / or B can represent: the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the related objects before and after are in an "or" relationship.

[0041] References to "one embodiment" or "some embodiments" etc. described in this specification mean that one or more embodiments of the present invention include the specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. appearing in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in another way. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in another way. The term "connected" includes direct and indirect connections, unless otherwise specified. "First" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated.

[0042] In the embodiments of the present invention, words such as "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present invention should not be construed as preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0043] The image deblurring method provided in the embodiment of the present invention can be applied to Figure 1 The application scenario shown includes a server 100 and a terminal device 200.

[0044] In one possible design, the server 100 is used to obtain a blurred image from the terminal device 200, then calculate the blurriness of the blurred image, and then select a corresponding deep learning network model based on the blurriness to perform a deblurring operation.

[0045] In another possible design, after the terminal device 200 generates a blurred image, it can first calculate the blurriness of the blurred image, and then select the corresponding deep learning network model to perform deblurring operation based on the blurriness, or use the corresponding deep learning network model in the server to perform deblurring operation.

[0046] The server 100 and the terminal device 200 can be connected via a wireless network. The terminal device 200 is a terminal device with network communication capabilities, such as a smart phone, a tablet computer, or a portable personal computer. The server 100 can be a single server, or a server cluster or a cloud computing center consisting of multiple servers.

[0047] based on Figure 1 As shown in the application scenario diagram, the embodiment of the present invention provides a process of an image deblurring method, such as Figure 2 As shown, the process of the method can be executed by the server 100 or the terminal device 200, and the following is illustratively described using the server 100 as an example. The method includes the following steps:

[0048] S301, extracting original image information of three channels from the blurred image.

[0049] S302 , performing Laplace transform and mean square error calculation on the original image information of the three channels respectively to obtain intermediate image information corresponding to the three channels respectively.

[0050] S303: Average the intermediate image information corresponding to the three channels to obtain the blurriness of the blurred image.

[0051] S304: Select a corresponding deep learning network model to perform a deblurring operation on the blurred image according to the blurriness of the blurred image.

[0052] Combination of the above methods Figure 3 For example, after the server 100 obtains the blurred image from the terminal device 200, it first extracts the original image information of the R channel, the original image information of the G channel, and the original image information of the B channel, and then performs Laplace transform and mean square error calculation on the original image information of the R channel to obtain the intermediate image information of the R channel; performs Laplace transform and mean square error calculation on the original image information of the G channel to obtain the intermediate image information of the G channel; performs Laplace transform and mean square error calculation on the original image information of the B channel to obtain the intermediate image information of the B channel. The intermediate image information corresponding to the three channels is averaged to obtain the blurriness of the blurred image. If the blurriness of the blurred image is greater than or equal to the set threshold, a gradually upsampling deep learning network model is used to perform a deblurring operation on the blurred image; if the blurriness of the blurred image is lower than the set threshold, a two-layer parallel deep learning network model is used to perform a deblurring operation on the blurred image.

[0053] It's important to note that this method, by using a smaller network model to estimate the image's blur level, can skip image preprocessing, significantly reducing computational overhead while preserving the original image information as much as possible. This compares to traditional deblurring algorithms, which first preprocess the image. To reduce computational overhead, the user must select an image block. The second step uses a Bayesian algorithm to estimate the blur kernel function, which can easily fall into local optima. Finally, a standard deblurring algorithm is used to reconstruct a clear image. This makes the entire process computationally complex, and often unsatisfactory results are difficult to achieve when the blur type is unknown or the blur level is high.

[0054] In one possible embodiment, as shown in FIG4 , the gradually upsampling deep learning network model provided in this embodiment includes: a first feature extraction branch 401, a second feature extraction branch 402, a fully connected module 403, and an image reconstruction branch 404, each feature extraction branch being composed of a different network model. The first feature extraction branch 401 is used to extract a first feature map from the blurred image; the second feature extraction branch 402 is used to extract a second feature map from the blurred image; the fully connected module 403 is used to fully connect the first feature map and the second feature map to obtain a third feature map;

[0055] The image reconstruction branch 404 is used to add the blurred image to the first feature map to obtain a first intermediate image of the next level; add the intermediate image to the second feature map to obtain a second intermediate image of the next level; and finally add the third feature map obtained after full connection to the second intermediate image to obtain a clear image.

[0056] Among them, the first feature extraction branch 401 draws on the residual structure in the ResNet network to obtain image features. It is composed of multiple different convolutional layers. Each level of convolutional layer obtains nonlinear feature mapping, and finally performs a transposed convolution operation on each layer to obtain the enhanced image features. Figure 4A 4041 represents the image feature model obtained by different convolutional layers. The image features are obtained by performing a transposed convolution operation on the feature model. The transposed convolution is used to better extract image features. The second feature extraction branch 402 draws on the VGGNet network block structure and is mainly used to extract image details.

[0057] in addition, Figure 4B Specifically shown is a feature extraction principle block diagram of the first feature extraction branch 401 and the second feature extraction branch 402. The extraction process uses an upsampling step, which mainly includes: convolution, feature embedding, feature upsampling and parameter sharing operations.

[0058] It is worth noting that the number of convolution layers can be one or more. In this embodiment, the convolution layer may include a plurality of convolution operators, which are also called kernels. Their role in image processing is equivalent to a filter that extracts specific information from an input image matrix. The convolution operator can essentially be a weight matrix, which is usually predefined. In the process of performing a convolution operation on an image, the weight matrix is usually processed one pixel by one pixel (or two pixels by two pixels) horizontally on the input image, thereby completing the work of extracting specific image features from the image.

[0059] In one possible embodiment, Figure 5As shown, for blurred images with a large degree of blur, the two-layer parallel deep learning network model designed by the present invention includes a first-layer deep learning network model and a second-layer deep learning network model; the first-layer deep learning network model is used to retain the color information and information of the image to obtain a first feature map; the first-layer deep learning network model is used for deblurring operation to obtain a second feature map; wherein the superposition result of the first feature map and the second feature map is equal to a clear image.

[0060] Among them, Figure 5 As shown, the first-layer deep learning network model extracts the three color channels and performs convolution operations on each. The features of the three color channels are then fused through concatenation. Finally, a simple feature extraction and fusion operation is performed to obtain a color feature map. The second-layer deep learning network model uses a common pyramid-like structure to retain useful information and remove blurry information, ultimately producing a feature map. The information from the first and second layers is added together to produce the final clear image.

[0061] Based on the above image deblurring model, in some embodiments of the present invention, an image deblurring device is disclosed, such as Figure 6 As shown, the device is used to implement the methods described in the above method embodiments, which includes: an extraction unit 601, a calculation unit 602, and a processing unit 603. Among them:

[0062] An extraction unit 601 is used to extract original image information of three channels from the blurred image;

[0063] The calculation unit 602 is configured to perform Laplace transform and mean square error calculation on the original image information of the three channels respectively to obtain intermediate image information corresponding to the three channels respectively; and average the intermediate image information corresponding to the three channels respectively to obtain the blur degree of the blurred image;

[0064] The processing unit 603 is used to select a corresponding deep learning network model to perform a deblurring operation on the blurred image according to the blurriness of the blurred image.

[0065] All relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0066] The image deblurring device may be a chip or a chip module. Regarding the various modules / units included in the various devices and products described in the above embodiments, they may be software modules / units, hardware modules / units, or partly software modules / units and partly hardware modules / units.

[0067] In other embodiments of the present invention, the present invention discloses an electronic device, such as Figure 7 As shown, the electronic device may include: one or more processors 701; a memory 702; a display 703; one or more application programs (not shown); and one or more computer programs 704. The above components may be connected via one or more communication buses 705. The one or more computer programs 704 are stored in the memory 702 and configured to be executed by the one or more processors 701. The one or more computer programs 704 include instructions.

[0068] The present invention also provides a computer-readable medium having a computer program stored thereon, which, when executed by a computer, implements the method described in the above method embodiment. Specific beneficial effects can be found in the above method embodiment.

[0069] The present invention also provides a computer program product, which, when executed by a computer, implements the method described in the above method embodiment. Specific beneficial effects can be found in the above method embodiment.

[0070] Through the description of the above embodiments, those skilled in the art will clearly understand that for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working processes of the above-described systems, devices, and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0071] Each functional unit in each embodiment of the present invention may be integrated into a processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The above-mentioned integrated units may be implemented in the form of hardware or software functional units.

[0072] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: flash memory, mobile hard disk, read-only memory, random access memory, magnetic disk or optical disk, etc., various media that can store program code.

[0073] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present invention should be covered by the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. An image deblurring method, characterized in that: The method includes: Extract the original image information of three channels from the blurred image; Perform Laplace transform and mean square error calculation on the original image information of the three channels respectively to obtain the intermediate image information corresponding to the three channels; Averaging the intermediate image information corresponding to the three channels to obtain the blurriness of the blurred image; According to the blurriness of the blurred image, a corresponding deep learning network model is selected to perform a deblurring operation on the blurred image, including: when the blurriness of the blurred image is greater than or equal to a set threshold, a gradually upsampling deep learning network model is used to perform a deblurring operation on the blurred image; when the blurriness of the blurred image is lower than the set threshold, a two-layer parallel deep learning network model is used to perform a deblurring operation on the blurred image; wherein, The gradually upsampling deep learning network model includes a first feature extraction branch, a second feature extraction branch, a fully connected module and an image reconstruction branch; the first feature extraction branch, drawing on the residual structure in the ResNet network, is composed of multiple different convolutional layers, each level of convolutional layer obtains a nonlinear feature map, and finally each layer is transposed convolution operation to extract the first feature map from the blurred image using the upsampling step; the second feature extraction branch, drawing on the VGGNet network block structure, is used to extract the second feature map from the blurred image using the upsampling step; the fully connected module is used to fully connect the first feature map and the second feature map to obtain a third feature map; the image reconstruction branch is used to add the blurred image and the first feature map to obtain a first intermediate image of the next level; the intermediate image and the second feature map are added to obtain a second intermediate image of the next level, and finally the third feature map obtained after full connection is added to the second intermediate image to obtain a clear image; The two-layer parallel deep learning network model includes a first-layer deep learning network model and a second-layer deep learning network model; the first-layer deep learning network model first extracts the three color channels, performs convolution operations on each of them, then fuses the features of the three color channels through a connection method, and finally further performs a feature extraction and fusion operation to obtain a first feature map of color information; the second-layer deep learning network model adopts a pyramid-like structure for deblurring operation to obtain a second feature map; the superposition result of the first feature map and the second feature map is equal to a clear image.

2. An image deblurring device, used to implement the image deblurring method according to claim 1, characterized in that: The device includes: An extraction unit, used for extracting original image information of three channels from the blurred image; a calculation unit, configured to perform Laplace transform and mean square error calculation on the original image information of the three channels respectively to obtain intermediate image information corresponding to the three channels respectively; and average the intermediate image information corresponding to the three channels respectively to obtain the blur degree of the blurred image; A processing unit is used to select a corresponding deep learning network model to perform a deblurring operation on the blurred image according to the blurriness of the blurred image, specifically for: when the blurriness of the blurred image is greater than or equal to a set threshold, using a gradually upsampling deep learning network model to deblur the blurred image; when the blurriness of the blurred image is lower than the set threshold, using a two-layer parallel deep learning network model to deblur the blurred image.

3. The device according to claim 2, characterized in that The gradually upsampling deep learning network model includes a first feature extraction branch, a second feature extraction branch, a fully connected module and an image reconstruction branch, each feature extraction branch is composed of a different network model. The first feature extraction branch is used to extract a first feature map from the blurred image; The second feature extraction branch is used to extract a second feature map from the blurred image; The fully connected module is used to fully connect the first feature map and the second feature map to obtain a third feature map; The image reconstruction branch is used to add the blurred image to the first feature map to obtain a first intermediate image of the next level; add the intermediate image to the second feature map to obtain a second intermediate image of the next level; and finally add the third feature map obtained after full connection to the second intermediate image to obtain a clear image.

4. The device according to claim 2, characterized in that The dual-layer parallel deep learning network model includes a first-layer deep learning network model and a second-layer deep learning network model; The first layer of deep learning network model is used to retain the color information and information of the image and obtain the first feature map; The first layer of deep learning network model is used for deblurring operation to obtain the second feature map; The superposition result of the first feature map and the second feature map is equal to a clear image.

5. A computer-readable storage medium having a computer program stored therein, characterized in that: When the computer program is executed by a processor, the method according to claim 1 is implemented.

6. An electronic device, characterized in that: The electronic device comprises a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the computer program is executed by the processor, the electronic device implements the method according to claim 1.

Images