Image Deblurring Method, Apparatus, Computer-Readable Medium, and Electronic Device

Abstract

The present disclosure provides an image deblurring method and apparatus, a computer-readable medium, and an electronic device, relating to the technical field of image processing. The method includes: obtaining a blurred original image collected by a target real lens; inputting the blurred original image into an image deblurring model associated with the target real lens to generate a clear original image; performing image signal processing on the clear original image and outputting a clear image. The present disclosure can perform deblurring in the RAW domain of an image, effectively improve the image quality of the clear image, improve the processing efficiency of image deblurring, and can realize real-time preview of the clear image.

Description

Background Art

[0002] With the continuous improvement of people's living standards, taking pictures and videos has become a common form of entertainment, and the improvement of image quality has attracted more and more attention. Generally, the image quality can be improved by performing deblurring processing on the image.

[0003] Currently, in related image deblurring solutions, generally, the image in the RGB domain is deblurred to improve the image quality. However, the image quality of the output image obtained by this technical solution is poor and the real-time performance is poor, and the real-time preview of the output image cannot be realized. Summary of the Invention

[0004] The purpose of the present disclosure is to provide an image deblurring method, an image deblurring device, a computer-readable medium, and an electronic device, so as to at least overcome to a certain extent the problems that the image quality of the output image in the related art is poor and the real-time preview of the output image cannot be realized.

[0005] According to the first aspect of the present disclosure, there is provided an image deblurring method, including:

[0006] Obtaining a blurred original image collected by a target real lens;

[0007] Inputting the blurred original image into an image deblurring model associated with the target real lens, and outputting a clear original image;

[0008] Performing image signal processing on the clear original image, and outputting a clear image.

[0009] According to the second aspect of the present disclosure, there is provided an image deblurring device, including:

[0010] An image acquisition module, configured to acquire a blurred original image collected by a target real lens;

[0011] An image deblurring module, configured to input the blurred original image into an image deblurring model associated with the target real lens, and output a clear original image;

[0012] An image output module, configured to perform image signal processing on the clear original image, and output a clear image.

[0013] According to the third aspect of the present disclosure, there is provided a computer-readable medium, on which a computer program is stored, and when the computer program is executed by a processor, the above method is implemented.

[0014] According to the fourth aspect of the present disclosure, there is provided an electronic device, characterized by including:

[0015] A processor; and

[0016] A memory for storing one or more programs, which when executed by one or more processors, cause the one or more processors to implement the above-mentioned method.

[0017] The image deblurring method provided by an embodiment of the present disclosure can obtain a blurred original image captured by a target real lens, then input the blurred original image into an image deblurring model associated with the target real lens to output a clear original image, and further perform image signal processing on the clear original image to output a deblurred clear image. On the one hand, directly obtaining the blurred original image in the RAW domain captured by the target real lens and performing deblurring processing on the blurred original image in the RAW domain through an image deblurring model pre-trained in the simulation training process can effectively improve the image quality of the output clear image and ensure the clarity of the output image. On the other hand, directly performing deblurring processing on the blurred original image that has not undergone any processing can avoid the process of waiting for the blurred original image in the RAW domain to be processed to obtain an image in the RGB domain in related technical solutions, effectively improving the efficiency of the image deblurring process, improving the real-time output of the clear image, and realizing real-time preview of the clear image. On the other hand, the deblurring process of the image can be jointly realized by combining the target real lens and the image deblurring model associated with the target real lens. While effectively improving the image quality of the output image, it is not necessary to adjust the lens structure of the target real lens or train and adapt the image deblurring model only when the target real lens is set in the terminal device, improving the matching between the target real lens and the image deblurring model, effectively reducing manpower and material resources, and saving hardware costs.

[0018] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings herein are incorporated into the specification and constitute a part of this specification, showing embodiments consistent with the present disclosure and used together with the specification to explain the principles of the present disclosure. Obviously, the accompanying drawings in the following description are only some embodiments of the present disclosure, and those of ordinary skill in the art can obtain other drawings without creative efforts based on these drawings. In the drawings:

[0020] Figure 1 A schematic diagram showing an exemplary system architecture to which the embodiments of the present disclosure can be applied;

[0021] Figure 2 A schematic flow diagram showing an image deblurring method in an exemplary embodiment of the present disclosure;

[0022] Figure 3Schematically shows a flowchart of implementing a simulation training process in an exemplary embodiment of the present disclosure;

[0023] Figure 4 Schematically shows a flowchart of implementing image conversion in an exemplary embodiment of the present disclosure;

[0024] Figure 5 Schematically shows a schematic diagram of the principle of image conversion through an image template in an exemplary embodiment of the present disclosure;

[0025] Figure 6 Schematically shows a schematic diagram of the structure of an image deblurring model in an exemplary embodiment of the present disclosure;

[0026] Figure 7 Schematically shows a flowchart of training an image deblurring model in an exemplary embodiment of the present disclosure;

[0027] Figure 8 Schematically shows a schematic diagram of the composition of an image deblurring device in an exemplary embodiment of the present disclosure;

[0028] Figure 9 Shows a schematic diagram of an electronic device to which the embodiments of the present disclosure can be applied. Detailed implementation manners

[0029] Example embodiments will now be described more fully with reference to the accompanying drawings. However, the example embodiments can be implemented in various forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be more complete and comprehensive, and will fully convey the concept of the example embodiments to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.

[0030] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus their repeated description will be omitted. Some of the block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0031] Figure 1 Shows a schematic diagram of the system architecture of an exemplary application environment of an image deblurring method and device to which the embodiments of the present disclosure can be applied.

[0032] As Figure 1As shown, the system architecture 100 may include one or more of terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc. The terminal devices 101, 102, 103 may be various electronic devices with image processing functions, including but not limited to portable computers, smart phones, smart surveillance cameras, and smart robots, etc. It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. According to the implementation requirements, there may be any number of terminal devices, networks and servers. For example, the server 105 may be a server cluster composed of multiple servers.

[0033] The image deblurring method provided in the embodiment of the present disclosure is generally executed by the terminal devices 101, 102, and 103, and accordingly, the image deblurring device is generally disposed in the terminal devices 101, 102, and 103. However, it is easy for those skilled in the art to understand that the image deblurring method provided in the embodiment of the present disclosure may also be executed by the server 105, and accordingly, the image deblurring device may also be disposed in the server 105, which is not particularly limited in the present exemplary embodiment.

[0034] For example, in an exemplary embodiment, a user may capture a blurred original image through an image sensor for capturing image information included in terminal devices 101, 102, and 103, and then upload the blurred original image to server 105. After the server generates a clear image through the image deblurring method provided in an embodiment of the present disclosure, the server transmits the clear image to terminal devices 101, 102, 103, etc. for output and display.

[0035] At present, whether it is a mobile terminal camera or a professional camera, the image quality of the images that can be captured is getting higher and higher, but with it comes an increasingly complex lens group design. However, such a design has caused the shooting equipment to continue to increase in size and weight, and the lens module design always has physical limits. Therefore, in terms of improving image quality, in addition to shooting a more complex lens group design, other aspects should also be considered.

[0036] Thanks to the recent popularity of Artificial Intelligence (AI) and its excellent performance in image restoration, it is possible to consider designing a lens with a simpler structure to capture an image with not-so-good quality, and then use AI technology to restore it into a clear image with high image quality. Generative Adversarial Networks (GAN) is one of the deep learning models commonly used for image restoration in recent years. The generative adversarial model can produce quite good outputs through the mutual game learning of the Generator and the Discriminator.

[0037] On the other hand, in the aspect of image deblurring, in related technologies, generally, deblurring processing is performed on images in the RGB domain, where the images in the RGB domain can be obtained by performing Image Signal Processing (ISP) on images in the RAW domain. Since the RAW information obtained after the acquisition and conversion by the camera is problematic and incomplete, after ISP, it may lead to the accumulation and mutual influence of various problems, making it difficult to obtain a good restoration effect when performing deblurring processing on images in the RGB domain.

[0038] The image data in the RAW domain is the most original data obtained by the lens module and has not been affected by the superposition of a series of operations of Image Signal Processing (IPS). Therefore, it is more reasonable to perform deblurring (Deblur) operations on the image data in the RAW domain first and then perform ISP. At the same time, in order to achieve the effect of real-time preview, image deblurring processing should also be performed in the RAW domain. If it goes through the ISP process, it will take a lot of time and it will be difficult to achieve real-time effects.

[0039] Based on one or more problems existing in related technologies, taking the execution of a terminal device as an example, the image deblurring method and the image deblurring device of the exemplary embodiments of the present disclosure will be specifically described below.

[0040] Figure 2 The flowchart of an image deblurring method in this exemplary embodiment is shown, including the following steps S210 to step S230:

[0041] In step S210, a blurred original image captured by a target real lens is obtained.

[0042] In an exemplary embodiment, a target real lens refers to a lens structure designed and manufactured in advance through simulation experiments. Multiple different types of real lenses can be designed and manufactured in advance through simulation experiments. During actual use, a target real lens that can be applied to the terminal device can be selected from multiple real lenses according to the function design of the terminal device (such as long-focus lens, night vision lens, wide-angle lens, etc.).

[0043] A blurred original image refers to an image captured by the target real lens that has not undergone any processing, that is, a RAW image. A RAW image is the raw data obtained by converting the light source signal captured by a CMOS image sensor or a CCD image sensor into a digital signal. A RAW file is a file that records the original information of a digital camera sensor and also records some metadata generated by the camera shooting (such as ISO settings, shutter speed, aperture value, white balance, etc.). RAW is an unprocessed and uncompressed format. The image data in the RAW domain can be simply understood as "original image coding data" or "digital negative film".

[0044] Optionally, when the target real lens captures a blurred original image, the blurred original image can be directly obtained; alternatively, the current image output for display, that is, the image in the RGB domain, can be obtained, and the blurred original image corresponding to the current image can be obtained through conversion processing of the current image in the RGB domain. The present exemplary embodiment does not make any special limitations on the method for obtaining the blurred original image.

[0045] In step S220, the blurred original image is input into an image deblurring model associated with the target real lens to generate a clear original image.

[0046] In an exemplary embodiment, an image deblurring model refers to a deep learning model used for deblurring image data in the RAW domain. For example, the image deblurring model can be a model for image deblurring constructed based on a generative adversarial network (GAN), or a model for image deblurring constructed based on a convolutional neural network (CNN). Of course, the image deblurring model can also be other types of deep learning models that can achieve image deblurring. The present exemplary embodiment does not make any special limitations on the type of deep learning model used by the image deblurring model.

[0047] A clear original image refers to the image data in the RAW domain obtained by deblurring the blurred original image in the RAW domain.

[0048] In step S230, image signal processing is performed on the clear original image to output a clear image.

[0049] In an exemplary embodiment, the Image Signal Processing (ISP) refers to the process of processing the RAW domain image data output by the front-end image sensor to obtain the image data for final display output. For example, the image signal processing may at least include Automatic Exposure Control (AEC), Automatic Gain Control (AGC), Automatic White Balance (AWB), color correction, Gamma correction, dead pixel removal, etc. The present exemplary embodiment is not limited thereto.

[0050] The blurred original image collected is deblurred by the image deblurring model to obtain a clear original image, and then the deblurred clear original image is processed according to the normal image signal processing flow to obtain a clear image with high image quality, which can effectively improve the image quality of the output clear image. Since it is not necessary to wait until the image signal processing of the RAW domain image data is performed before deblurring, the deblurring processing efficiency of the image is effectively improved, and real-time preview of the clear image is realized.

[0051] Next, the content in steps S220 to S230 will be described in detail.

[0052] In an exemplary embodiment, the image deblurring model in step S220 can be obtained through a simulation training process. The simulation training process refers to the process of training a pre-constructed model according to a sample data set to obtain a trained image deblurring model. Among them, the target real lens can be constructed based on the target simulation lens data in the simulation training process.

[0053] Specifically, it can be achieved through the steps in Figure 3 to implement the simulation training process of the image deblurring model. As shown in Figure 3 , the simulation training process may specifically include:

[0054] Step S310, obtaining a first clear image, and determining a first blurred image based on the first clear image and pre-designed simulation lens data;

[0055] Step S320, determining a first clear original image corresponding to the first clear image, and determining a first blurred original image corresponding to the first blurred image;

[0056] Step S330, training the pre-constructed simulation basic model based on the first clear original image and the first blurred original image to generate a simulation image deblurring model;

[0057] Step S340: Obtain a second clear image, and determine a second blurred image based on the second clear image and a real lens constructed based on the simulation lens data;

[0058] Step S350: Determine the second clear original image corresponding to the second clear image, and determine the second blurred original image corresponding to the second blurred image;

[0059] Step S360: Continuously perform network training on the simulation image deblurring model according to the second clear original image and the second blurred original image to generate an image deblurring model associated with the real lens.

[0060] Among them, the first clear image refers to a pre-collected set of clear sample RGB-domain images. The first clear image can use RGB images of the Adobe5K standard. Of course, it can also be RGB images with the same depth as the Adobe5K standard and similar sizes. This exemplary embodiment is not limited thereto.

[0061] The simulation lens data refers to a simulation method for simulating the collection of optical signals by a real lens. For example, the simulation lens data can be a point spread function (PSF) constructed based on the designed lens module structure, or a convolutional network constructed based on the designed lens module structure. Of course, the simulation lens data can also be other types of simulation methods that can be used to simulate the collection of optical signals by a real lens. This exemplary embodiment does not make special limitations on this.

[0062] The first blurred image refers to a set of blurred sample RGB-domain images obtained by converting the first clear image through the simulation lens data. For example, taking the simulation lens data as the constructed point spread function PSF as an example, the first clear image can be convolved through the point spread function to obtain the first blurred image corresponding to the first clear image. By specifying the simulation lens data of the lens design and an easy-to-find set of clear RGB-domain images, a set of simulated RAW-domain images for training can be obtained, without specifically looking for a set of simulated RAW-domain images, effectively reducing the difficulty of sample collection.

[0063] Since the image deblurring model needs to deblur images from the RAW domain during application, during training, the first clear image and the first blurred image also need to be converted to the RAW domain. Specifically, it can be achieved through the steps in Figure 4 to convert the first clear image and the first blurred image. As shown in Figure 4 it specifically includes:

[0064] Step S410: Obtain a preset image template;

[0065] Step S420, perform dot extraction on the first clear image through the image template to obtain a first clear original image; and

[0066] Step S430, perform dot extraction on the first blurred image through the image template to obtain a first blurred original image.

[0067] Among them, the image template refers to a pre-specified Bayer template (Bayer Pattern) used to convert images in the RGB domain. For example, the image template can be an RGGB Bayer template or an RGBG Bayer template, and can be specifically customized according to actual usage. This exemplary embodiment does not make special limitations on this.

[0068] The dot extraction can be performed on the first clear image in the RGB domain through the image template to obtain the first clear original image in the RAW domain. Similarly, the dot extraction can be performed on the first blurred image in the RGB domain through the image template to obtain the first blurred original image in the RAW domain.

[0069] Reference Figure 5 As shown, for the first clear image or the first blurred image 501, a pre-specified image template 502 can be obtained. Since the human eye is more sensitive to green, an image template with G occupying two grids and R and B each occupying only one grid can be taken. That is, the image template 502 can be an RGGB template, which can be specifically shown as Figure 5 the 2*2 grid shown in ; by covering the entire first clear image or the first blurred image 501 in the RGB domain with the image template 502, taking out the channel values of the image template 502 corresponding to each pixel, and obtaining the first clear original image or the first blurred original image 503 in the single-channel RAW domain. Of course, Figure 5 This is only a schematic example and should not cause any special limitations to this exemplary embodiment.

[0070] By the method of performing dot extraction on the image through the specified image template, the conversion of the RGB domain image to the RAW domain image is realized. After verification, after the converted RAW domain image is restored to the RGB domain image through the demosaic algorithm (Demosaic), it has a high similarity with the original RGB domain image, indicating that the method of performing dot extraction on the RGB domain image according to the image template is reasonable and effective.

[0071] Continue to refer to Figure 3As shown, the simulation basic model in step S330 refers to a pre-constructed deep learning model that has not yet started training. For example, the simulation basic model can be a model for image deblurring constructed based on the generative adversarial network (GAN), or a model for image deblurring constructed based on the convolutional neural network (CNN). This exemplary embodiment is not limited thereto. After the simulation basic model undergoes the Figure 3 network training process in

[0072] it, a trained image deblurring model can be obtained. That is, the network structures of the simulation basic model and the image deblurring model are the same.

[0073] Optionally, before using the first clear original image and the first blurred original image as a sample data set to perform network training on the simulation basic model, data augmentation can be performed on the first clear original image and the first blurred original image. For example, after cropping a training segment of a specified size (such as 512×512), rotation and / or mirroring operations can be performed on the cropped training segment based on a specified image template, so that the rotated and / or mirrored training segment still conforms to the previously specified image template, in order to enhance the training effect of the model with the data-augmented first clear original image and first blurred original image.

[0074] After training is completed to obtain the simulation image deblurring model, the simulation image deblurring model can be evaluated. If the model evaluation result meets the requirements for the image restoration effect, it can be considered that the design of the simulation simulation lens corresponding to the simulation lens data is reasonable and can be used for actual applications. Therefore, a real lens can be constructed according to the simulation lens data.

[0075] At the same time, during the actual application process, due to the gap between the actual process and theory, if the images captured by the real lens are directly used with the simulation image deblurring model trained under simulation conditions, the image deblurring effect is poor. Therefore, it is necessary to further optimize the simulation image deblurring model according to the actual shooting situation of the real lens.

[0076] Continue to refer to Figure 3 As shown, the second clear image in step 340 refers to a pre-collected set of clear sample RGB domain images. It can be understood that the second clear image can continue to use the first clear image, or other clear images other than the first clear image can be obtained as the second clear image. This exemplary embodiment does not make special limitations on this.

[0077] The real lens refers to a lens module constructed based on the designed simulation lens data.

[0078] The second blurred image refers to a set of blurred sample RGB domain images obtained by photographing the second clear image with the real lens. For example, the real lens can be fixed and faced to the screen displaying the second clear image, and the second clear image displayed on the screen can be automatically replaced and photographed through an automated script to obtain the second blurred image in the RGB domain. With the made real lens and an easy-to-find set of clear RGB domain images, a real RAW domain image set for training can be obtained without specifically looking for a real RAW domain image set, effectively reducing the difficulty of sample collection.

[0079] It is easy for those skilled in the art to understand that reference can be made to Figure 4 、 Figure 5 the process of image conversion for the first clear image and the first blurred image in, and dot extraction processing is performed on the second clear image and the second blurred image to obtain the second clear original image and the second blurred original image.

[0080] Specifically, the second clear original image and the second blurred original image can continue to be input into the trained simulation image deblurring model for further network training. If necessary, the loss function and training hyperparameters and other data of the simulation image deblurring model can be adjusted until the clear original image output by the simulation image deblurring model meets the preset conditions (such as the preset condition can be that the similarity between the clear original image output by the simulation basic model and the second clear original image is greater than the similarity threshold, or the loss between the clear original image output by the simulation basic model and the second clear original image converges), and a trained image deblurring model is obtained. Combining this image deblurring model with the real lens can output a relatively clear image even when the real lens has a simple structure, improving the image quality and clarity of the output image. On the other hand, training first on the simulation dataset and then manufacturing the real lens model for training on the real dataset after achieving good results can save a lot of time, reduce costs, and effectively shorten the training cycle of the image deblurring model.

[0081] It can be understood that the simulation basic model, the simulation image deblurring model, or the image deblurring model in this exemplary embodiment are essentially different adjustment and training stages of the same deep learning model. Specifically, referring to Figure 6 as shown, the simulation basic model, the simulation image deblurring model, or the image deblurring model can be constructed by the WGAN-GP model 600 based on the Unet network structure. The WGAN-GP model 600 based on the Unet network structure can at least include a Generator 601 and a Discriminator 602. Among them, the Generator 601 is of the UNet (intuitively going down first and then up, shaped like a U, so it is called a U-shaped network) structure, which contains 6 pairs of upsampling and downsampling network structures. The Generator 601 uses the Instance Normalization (IN) algorithm. The IN algorithm is a normalization algorithm that is more suitable for scenarios with higher requirements for individual pixels. And the network structure of the Generator 601 uses Skip Connections. Skip Connections refer to skipping some layers in the neural network and using the output of one layer as the input of the next layer to ensure the reusability of features, make the loss of the model smoother, and effectively improve the convergence speed of the network.

[0082] The loss function of the Generator 601 can include Perceptual loss (by default, using the VGG model, and the VGG model is a model for extracting features from images) and WGAN Loss, while the loss function of the Discriminator 602 can be the Wasserstein Distance.

[0083] Before calculating the perceptual error, the single-channel RAW domain image data can be copied three times and combined into three-channel data for the feature map operation of the pre-trained VGG model.

[0084] Of course, Figure 6 the model structure in

[0085] Optionally, when training a network on the simulation basic model to obtain a simulation image deblurring model, or when training a network on the simulation image deblurring model to obtain an image deblurring model, the proportion of each loss function or each sub-component of each loss function (for example, the VGG that extracts image features in the generator 601 can include the Content Loss of the feature map and the Style Loss that calculates the Gram matrix in the channel dimension of the feature map) can be adjusted, and by adjusting the size of hyperparameters, for example, the hyperparameters can be Epoch, Batch Size, etc., to debug to meet the requirements of the image restoration effect, so as to obtain a trained simulation image deblurring model or an image deblurring model.

[0086] In an exemplary embodiment, before inputting the second clear original image and the second blurred original image into the simulation image deblurring model for continued network training, the simulation image deblurring model can be optimized and pruned to obtain a simplified simulation image deblurring model.

[0087] Since the trained simulation image deblurring model often has redundant structure and is too large in scale to be put into actual use, certain optimization and pruning are required.

[0088] Specific optimization and pruning methods can include general optimization and pruning methods and advanced optimization and pruning methods. Among them, general optimization and pruning methods can include reducing NGF (i.e., the reference value for the change in the number of generator channels), removing the normalization layer, changing the activation function, reducing the number of upsampling and downsampling operations and dimensional transformation of the Unet network to reduce the amount of computation, etc.; advanced optimization and pruning methods can include quantization training and knowledge distillation, etc. Among them, quantization training refers to converting the default float32 data into a smaller float16 or even integer int8 type, respectively compressing the space occupied by the model to 50% and 25% of the original, reducing the size of the model. Of course, after changing to a type with reduced precision, the effect of the model will be slightly affected. If the effect is still within an acceptable range, it is a successful optimization and pruning; knowledge distillation refers to training a teacher network with a complex structure and constructing a student network with a simple structure, and the student network migrates and learns the content of the teacher network to achieve optimization and pruning of the model.

[0089] Figure 7 Schematically shows a flowchart of training an image deblurring model in an exemplary embodiment of the present disclosure.

[0090] Refer to Figure 7 As shown, in step S710, a first clear RGB image is obtained, and the first clear RGB image is subjected to simulation lens PSF convolution processing to obtain a first blurred RGB image corresponding to the first clear RGB image;

[0091] Step S720: Obtain a specified image template (such as RGGB), and perform dot extraction on the first clear RGB image and the first blurred RGB image respectively according to this image template to obtain the first clear RAW image and the first blurred RAW image;

[0092] Step S730: Data augmentation can be performed on the first clear RAW image and the first blurred RAW image, and the first clear RAW image and the first blurred RAW image after data augmentation processing are input into a pre-constructed simulation basic model for network training to obtain a trained simulation basic model, that is, a simulation image deblurring model;

[0093] Step S740: Optimize and prune the simulation image deblurring model to obtain a simplified simulation image deblurring model;

[0094] Step S750: Obtain a second clear RGB image, and perform real lens shooting processing on the second clear RGB image to obtain a second blurred RGB image corresponding to the second clear RGB image;

[0095] Step S760: Obtain a specified image template (such as RGGB), and perform dot extraction on the second clear RGB image and the second blurred RGB image respectively according to this image template to obtain the second clear RAW image and the second blurred RAW image;

[0096] Step S770: Input the second clear RAW image and the second blurred RAW image into the simplified simulation image deblurring model to continue network training to obtain a trained image deblurring model. This image deblurring model combines with a real lens to achieve deblurring processing of the captured image, effectively improving the image quality and clarity of the output image, and enabling real-time preview of the clear image after deblurring.

[0097] In an application scenario, multiple simulation lens modules with a simple lens structure can be pre-designed, and the simulation lens data corresponding to the multiple simulation lens modules can be determined; then, according to different simulation lens data, in accordance with Figure 3Steps S310 to S330 in it are used to perform network training on the simulation basic model. If the sample data constructed based on a certain simulation lens data can complete the training of the simulation basic model and the model evaluation result of the simulation basic model meets the expected value, then a real lens is constructed according to the simulation lens data for subsequent network training; if the sample data constructed based on a certain simulation lens data cannot complete the training of the simulation basic model and the model evaluation result of the simulation basic model always fails to reach the expected value, then it can be considered that the simulation lens module corresponding to the simulation lens data is unreasonable and there is no need to construct the corresponding real lens. Through the simulation training process, multiple applicable real lenses can be quickly screened out, and an image deblurring model matching each real lens can be trained, which can save a lot of time and R & D costs. Moreover, through the combination of the real lens and the matching image deblurring model, while effectively simplifying the structure of the real lens, the image quality of the output image can be guaranteed; on the other hand, the target real lens and the image deblurring model associated with the target real lens can be combined to jointly achieve image deblurring processing. While ensuring the image quality of the clear image, there is no need to adjust the lens structure of the target real lens or train and adapt the image deblurring model only when the target real lens is set in the terminal device, which improves the matching between the target real lens and the image deblurring model, effectively reduces manpower and material resources, and saves hardware costs.

[0098] In an application scenario, after obtaining the trained image deblurring model, on this basis, an image data set under specific circumstances can be made to continue training to obtain an image deblurring model with a specific application scenario. For example, images in two special situations, strong light irradiation during the day and weak light at night, can be selected to continue training the image deblurring model respectively, and then two special modes can be obtained, that is, the strong light mode has a better effect when used in strong light, and the night mode has a better effect when used at night.

[0099] In summary, in the present exemplary embodiment, a blurred original image captured by a target real lens can be obtained, and then the blurred original image can be input into an image deblurring model associated with the target real lens to output a clear original image. Furthermore, image signal processing can be performed on the clear original image to output a deblurred clear image. On the one hand, directly obtaining the blurred original image in the RAW domain captured by the target real lens and performing deblurring processing on the blurred original image in the RAW domain through an image deblurring model pre-trained in the simulation training process can effectively improve the image quality of the output clear image and ensure the clarity of the output image. On the other hand, directly performing deblurring processing on the blurred original image that has not undergone any processing can avoid the process of waiting for the blurred original image in the RAW domain to be processed to obtain an image in the RGB domain in related technical solutions, effectively improving the efficiency of the image deblurring process, enhancing the real-time output of the clear image, and realizing the real-time preview of the clear image.

[0100] It should be noted that the above-mentioned drawings are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, rather than for limiting purposes. It is easy to understand that the processes shown in the above-mentioned drawings do not indicate or limit the time sequence of these processes. Additionally, it is also easy to understand that these processes can be executed synchronously or asynchronously in, for example, multiple modules.

[0101] Furthermore, referring to Figure 8 as shown, in the embodiment of the present example, an image deblurring device 800 is further provided, which may include an image acquisition module 810, an image deblurring module 820, and an image output module 830. Among them:

[0102] The image acquisition module 810 is configured to acquire a blurred original image captured by a target real lens;

[0103] The image deblurring module 820 is configured to input the blurred original image into an image deblurring model associated with the target real lens to generate a clear original image;

[0104] The image output module 830 is configured to perform image signal processing on the clear original image to output a deblurred clear image.

[0105] In an exemplary embodiment, the image deblurring model can be obtained through a simulation training process, and the target real lens can be constructed based on the target simulation lens data in the simulation training process; the image deblurring module 820 may include a simulation training unit, and the simulation training unit may include:

[0106] A first image acquisition sub-unit, configured to acquire a first clear image and determine a first blurred image based on the first clear image and pre-designed simulation lens data;

[0107] The first image conversion subunit is configured to determine a first clear original image corresponding to the first clear image and a first blurred original image corresponding to the first blurred image;

[0108] The simulation image deblurring model training subunit is configured to perform network training on a pre-constructed simulation basic model according to the first clear original image and the first blurred original image to generate a simulation image deblurring model;

[0109] The second image acquisition subunit is configured to acquire a second clear image and determine a second blurred image through the second clear image and a real lens constructed based on the simulation lens data;

[0110] The second image conversion subunit is configured to determine a second clear original image corresponding to the second clear image and a second blurred original image corresponding to the second blurred image;

[0111] The image deblurring model training subunit is configured to continue performing network training on the simulation image deblurring model according to the second clear original image and the second blurred original image to generate an image deblurring model associated with the real lens.

[0112] In an exemplary embodiment, the simulation lens data may include a point spread function corresponding to the simulation lens, and the first image acquisition subunit may be configured to:

[0113] Perform convolution processing on the first clear image based on the point spread function to obtain a first blurred image corresponding to the first clear image.

[0114] In an exemplary embodiment, the second image acquisition subunit may be configured to

[0115] Perform shooting processing on the second clear image based on the real lens to obtain a second blurred image corresponding to the second clear image.

[0116] In an exemplary embodiment, the image deblurring device 800 may further include a simulation image deblurring model pruning unit, and the simulation image deblurring model pruning unit may be configured to:

[0117] Perform optimization pruning processing on the simulation image deblurring model to obtain a simplified simulation image deblurring model.

[0118] In an exemplary embodiment, the first image conversion subunit may be configured to:

[0119] Obtain a preset image template;

[0120] Perform dot extraction processing on the first clear image through the image template to obtain a first clear original image; and

[0121] Perform dot extraction on the first blurred image through the image template to obtain a first blurred original image.

[0122] In an exemplary embodiment, the image deblurring model can be constructed by a WGAN-GP model based on the Unet network structure.

[0123] The specific details of each module in the above device have been described in detail in the implementation manner of the method part. For the undisclosed details, reference can be made to the implementation manner content of the method part, so it will not be elaborated here.

[0124] Those skilled in the art can understand that various aspects of the present disclosure can be implemented as a system, a method, or a program product. Therefore, various aspects of the present disclosure can be specifically implemented in the following forms, namely: a complete hardware implementation manner, a complete software implementation manner (including firmware, microcode, etc.), or an implementation manner combining hardware and software aspects, which can be collectively referred to as "circuit", "module", or "system" here.

[0125] An exemplary embodiment of the present disclosure provides an electronic device for implementing an image deblurring method, which can be Figure 1 the terminal devices 101, 102, 103 or the server 105 in. The electronic device includes at least a processor and a memory. The memory is used to store executable instructions of the processor, and the processor is configured to execute the image deblurring method by executing the executable instructions.

[0126] Next, take the Figure 9 electronic device 900 in as an example to make an exemplary description of the structure of the electronic device in the present disclosure. Figure 9 The shown electronic device 900 is only an example and should not bring any limitation to the functions and usage scope of the embodiments of the present disclosure.

[0127] As Figure 9 shown, the electronic device 900 is presented in the form of a general computing device. The components of the electronic device 900 may include but are not limited to: at least one processing unit 910, at least one storage unit 920, a bus 930 connecting different system components (including the storage unit 920 and the processing unit 910), and a display unit 940.

[0128] Among them, the storage unit 920 stores program codes, and the program codes can be executed by the processing unit 910, so that the processing unit 910 executes the motion posture determination method in this specification.

[0129] The storage unit 920 may include a readable medium in the form of a volatile storage unit, such as a random access memory (RAM) 921 and / or a cache storage unit 922, and may further include a read-only memory (ROM) 923.

[0130] The storage unit 920 may also include a program / utility 924 having a set (at least one) of program modules 925. Such program modules 925 include, but are not limited to: an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment.

[0131] The bus 930 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus structures.

[0132] The electronic device 900 may also communicate with one or more external devices 970 (such as sensor devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 900, and / or may communicate with any device that enables the electronic device 900 to communicate with one or more other computing devices (such as routers, modems, etc.). Such communication may be through an input / output (I / O) interface 950. Moreover, the electronic device 900 may also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 960. As shown in the figure, the network adapter 960 communicates with other modules of the electronic device 900 through the bus 930. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 900, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, data backup storage systems, and sensor modules (such as gyroscope sensors, magnetic sensors, acceleration sensors, distance sensors, proximity light sensors, etc.).

[0133] Through the description of the above embodiments, those skilled in the art can easily understand that the exemplary embodiments described herein can be implemented by software, or can be implemented by a combination of software and necessary hardware. Therefore, the technical solutions according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.

[0134] Exemplary embodiments of the present disclosure also provide a computer-readable storage medium having stored thereon a program product capable of implementing the methods described above in this specification. In some possible embodiments, various aspects of the present disclosure may also be implemented in the form of a program product, which includes program code. When the program product runs on a terminal device, the program code is used to cause the terminal device to execute the steps according to various exemplary embodiments of the present disclosure described in the "Exemplary Methods" section above, for example, it may execute Figures 2 to 7 any one or more of the steps.

[0135] It should be noted that the computer-readable medium shown in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0136] In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. And in the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination of the above.

[0137] In addition, program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and also including conventional procedural programming languages such as the "C" language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0138] Other embodiments of the present disclosure will be readily apparent to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include known common knowledge or conventional technical means in the technical field not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of the present disclosure are pointed out by the claims.

[0139] It should be understood that the present disclosure is not limited to the exact structures described above and shown in the figures, and various modifications and changes may be made without departing from its scope. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image deblurring method, characterized in that, Including: Obtain a blurred original image collected by a target real lens; Input the blurred original image into an image deblurring model associated with the target real lens to generate a clear original image; Perform image signal processing on the clear original image and output a clear image; The image deblurring model is obtained through a simulation training process, and the simulation training process includes: Obtain a first clear image, and determine a first blurred image based on the first clear image and pre-designed simulation lens data; Determine a first clear original image corresponding to the first clear image, and determine a first blurred original image corresponding to the first blurred image; Perform network training on a pre-constructed simulation basic model according to the first clear original image and the first blurred original image to generate a simulation image deblurring model; Obtain a second clear image, and determine a second blurred image based on the second clear image and a real lens constructed based on the simulation lens data; Determine a second clear original image corresponding to the second clear image, and determine a second blurred original image corresponding to the second blurred image; Continue to perform network training on the simulation image deblurring model according to the second clear original image and the second blurred original image to generate an image deblurring model associated with the real lens.

2. The method according to claim 1, wherein The target real lens is constructed based on target simulation lens data in the simulation training process.

3. The method according to claim 2, wherein The simulation lens data includes a point spread function corresponding to the simulation lens; the determining of the first blurred image based on the first clear image and the pre-designed simulation lens data includes: Perform convolution processing on the first clear image based on the point spread function to obtain a first blurred image corresponding to the first clear image.

4. The method according to claim 2, wherein The determining of the second blurred image based on the second clear image and the real lens constructed based on the simulation lens data includes: Perform shooting processing on the second clear image based on the real lens to obtain a second blurred image corresponding to the second clear image.

5. The method according to claim 2, wherein The method further includes: Perform optimization pruning processing on the simulation image deblurring model to obtain a simplified simulation image deblurring model.

6. The method according to claim 2, characterized in that The determining of the first clear original image corresponding to the first clear image, and the determining of the first blurred original image corresponding to the first blurred image includes: Obtain a preset image template; Perform dot extraction processing on the first clear image through the image template to obtain a first clear original image; and Perform dot extraction processing on the first blurred image through the image template to obtain a first blurred original image.

7. The method according to claim 1 or 2, characterized in that The image deblurring model is constructed by a WGAN-GP model based on a Unet network structure.

8. An image deblurring device, characterized in that, Including: An image acquisition module, configured to obtain a blurred original image collected by a target real lens; An image deblurring module, configured to input the blurred original image into an image deblurring model associated with the target real lens to generate a clear original image; An image output module, configured to perform image signal processing on the clear original image and output a clear image; The image deblurring model is obtained through a simulation training process, and the simulation training process includes: Obtain a first clear image, and determine a first blurred image based on the first clear image and pre-designed simulation lens data; Determine a first clear original image corresponding to the first clear image, and determine a first blurred original image corresponding to the first blurred image; Perform network training on a pre-constructed simulation basic model according to the first clear original image and the first blurred original image to generate a simulation image deblurring model; Obtain a second clear image, and determine a second blurred image based on the second clear image and a real lens constructed based on the simulation lens data; Determine a second clear original image corresponding to the second clear image, and determine a second blurred original image corresponding to the second blurred image; Continue to perform network training on the simulation image deblurring model according to the second clear original image and the second blurred original image to generate an image deblurring model associated with the real lens.

9. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method according to any one of claims 1 to 7.

10. An electronic device, characterized in that, Including: A processor; And A memory for storing executable instructions of the processor; Wherein, the processor is configured to execute the method according to any one of claims 1 to 7 by executing the executable instructions.

Images