Method for enhancing a blurred image and electronic device

Abstract

The application provides a method for enhancing a blurred image and an electronic device. The method comprises: generating N blurred images with different resolutions based on a blurred image, and inputting the N blurred images into an image enhancement model to calculate an output clear image. The image enhancement model is learned based on a plurality of groups of training images and through the following stages, each group of training images comprising a clear image I and corresponding N blurred images y i , 1≤i≤N: 1, learning Codebook and Decoder based on the clear image I in the training image and a first loss function; 2, learning Transformer based on all training images and a second loss function; 3, learning Encoder, a multi-resolution fusion module and CFT based on all training images and a third loss function, thereby obtaining the image enhancement model.

Description

Technical Field

[0001] This application relates to the field of terminals, and more particularly to a method and electronic device for enhancing blurred images. Background Technology

[0002] Currently, due to limitations imposed by various factors such as shooting equipment and shooting environment, images captured by electronic devices often suffer from blurriness. Mathematically, a blurry image captured by an electronic device is equivalent to the image obtained by convolving a sharp image with a blur kernel. However, in actual image capture, the causes of blurriness are complex and varied, meaning the blur kernel is unknown. Therefore, how to determine the unknown sharp image based on a known blurry image when both the blur kernel and the sharp image are unknown is a problem that urgently needs to be solved. Summary of the Invention

[0003] This application provides a method and electronic device for enhancing blurred images. The method includes: generating N blurred images with different resolutions based on a blurred image; inputting the N blurred images into an image enhancement model to calculate and output a sharp image. The image enhancement model is learned based on multiple sets of training images through the following stages: each set of training images includes one sharp image I and corresponding N blurred images y with different resolutions. i For each training image (1≤i≤N): 1. Learn the Codebook and Decoder based on the clear image I and the first loss function; 2. Learn the Transformer based on all training images and the second loss function; 3. Learn the Encoder, multi-resolution fusion module, and CFT based on all training images and the third loss function. This results in an image augmentation model containing the Encoder, Transformer, Codebook, Decoder, multi-resolution fusion module, and CFT.

[0004] In a first aspect, this application provides a method for enhancing blurred images. The method includes: acquiring N blurred images, wherein the N blurred images have different resolutions but the same blur features; using the N blurred images as input data for an image enhancement model, and outputting a corresponding first clear image; wherein the image enhancement model is learned based on multiple sets of training images and a loss function, each set of training images including: N training blurred images with the same resolution as the N blurred images and a corresponding training clear image; the loss function includes a temporal loss function and a frequency domain loss function.

[0005] Implementing the method described in the first aspect allows for the enhancement of low-quality, blurry images into high-quality, sharp images using an image enhancement model. Furthermore, because the image enhancement model is learned based on N images of different resolutions and a newly added time-frequency loss function, leveraging the invariant blur characteristics of multi-resolution images, a more accurate image enhancement model can be developed.

[0006] In conjunction with the method described in the first aspect, the image enhancement model includes an encoder, a converter, a codebook, and a decoder. Using the N blurred images as input data, the model outputs a corresponding first clear image. Specifically, this includes: encoding the N blurred images into N sets of corresponding feature vectors using the encoder; indexing the N sets of feature vectors using the converter and mapping them to codewords in the codebook; and outputting the first clear image based on the indexed codewords using the decoder. Here, the encoder is the same as the Encoder described in later embodiments, the converter is the same as the Transformer described in later embodiments, the codebook is the same as the Codebook described in later embodiments, and the decoder is the same as the Decoder described in later embodiments.

[0007] In conjunction with the method described in the first aspect, the image enhancement model includes an encoder, a converter, a codebook, a decoder, and N controllable feature transformation (CFT) modules. Using the N blurred images as input data for the image enhancement model, the corresponding first clear image is output. Specifically, this includes: encoding the N blurred images into N sets of corresponding feature vectors using the encoder; indexing the N sets of feature vectors using the converter and mapping them to N sets of codewords in the codebook; outputting N sets of intermediate feature vectors using the decoder based on the indexed N sets of codewords; adjusting the N sets of intermediate feature vectors using the N CFT modules; and outputting the first clear image using the decoder based on the adjusted N sets of intermediate feature vectors.

[0008] In this way, adding CFT can strike a balance between enhancing image quality and preserving the realism of the original image, thereby improving the performance of blurred image enhancement models.

[0009] In conjunction with the method described in the first aspect, the image enhancement model further includes N multi-resolution fusion modules. These N CFTs adjust the N sets of intermediate feature vectors based on the N sets of feature vectors. Specifically, this includes: fusing the N sets of feature vectors into N sets of fused feature vectors using the N multi-resolution fusion modules, with each set of fused feature vectors corresponding to the resolution of the N blurred images; and adjusting the N sets of intermediate feature vectors based on the N sets of fused feature vectors using the N CFTs. Specifically, one CFT is used to adjust one set of intermediate feature vectors based on one set of fused feature vectors.

[0010] In conjunction with the method described in the first aspect, the loss function also includes an index classification loss function; the encoder and the N CFTs are learned based on the blurred and sharp images in the training images, as well as the temporal loss function, the frequency domain loss function, and the index classification loss function.

[0011] In this way, by fully taking into account the various losses that exist in the process of learning the encoder and CFT, various loss functions are used to optimize the learning process in reverse, thereby learning a more accurate encoder and CFT.

[0012] In conjunction with the method described in the first aspect, the loss function also includes a codebook loss function and an adversarial network (GAN) loss function; the codebook is learned based on a clear image in the training image, as well as the temporal loss function, the frequency domain loss function, the codebook loss function, and the GAN loss function.

[0013] In this way, by fully taking into account the various losses that exist in the process of learning the codebook, various loss functions are used to optimize the learning process in reverse, thereby learning a more accurate codebook.

[0014] In conjunction with the method described in the first aspect, the loss function also includes an index classification loss function; the converter is learned based on the blurred and sharp images in the training images, as well as the temporal loss function, the frequency domain loss function, and the index classification loss function.

[0015] In this way, by fully taking into account the various losses that exist in the process of learning the converter, various loss functions are used to optimize the learning process in reverse, thereby learning a more accurate converter.

[0016] In conjunction with the method described in the first aspect, the encoder includes N residual blocks, and the decoder includes N residual blocks; the N residual blocks of the encoder are used to encode the N blurred images into corresponding N sets of feature vectors respectively; the first residual block of the decoder is used to receive the output of the other residual blocks and output the first clear image.

[0017] In this way, residual blocks connected by residuals can solve the problems of gradient vanishing and gradient exploding in the encoding process.

[0018] In conjunction with the method described in the first aspect, obtaining N blurred images specifically includes: obtaining a first blurred image, which has a first resolution; downsampling the first blurred image to obtain N-1 blurred images, which have different resolutions and the resolutions of the N-1 blurred images are less than the first resolution.

[0019] In this way, multiple blurred images of different resolutions can be generated based on the first blurred image to be processed, for example by downsampling, providing multiple data inputs for the deblurring process.

[0020] In conjunction with the method described in the first aspect, the first blurred image is one or more of the following types of images: images of people, images of animals, and images of landscapes.

[0021] In this way, the method provided in this application can be used for different types of images, thus expanding the application scenarios.

[0022] In conjunction with the method described in the first aspect, the blurriness of the first blurred image is greater than a preset value.

[0023] This eliminates the need for additional image enhancement processing on clear images, reducing the power consumption of electronic devices.

[0024] In conjunction with the method described in the first aspect, obtaining the first blurred image specifically includes: capturing the first blurred image using a camera.

[0025] This allows for direct image enhancement of images captured by electronic devices.

[0026] Combining the method described in the first aspect, N=3. Thus, image enhancement can be performed based on three blurred images with different resolutions, thereby obtaining a more accurate and clear image.

[0027] In a second aspect, this application provides an electronic device including one or more memories, one or more processors, and a computer program stored on the memory, the processor executing the computer program to implement the steps of the method described in any of the first aspects.

[0028] Thirdly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps of the method as described in any of the first aspects.

[0029] Fourthly, this application provides a computer program product including a computer program that, when executed by a processor, performs the steps of the method as described in any of the first aspects. Attached Figure Description

[0030] Figure 1 A schematic flowchart illustrating the blurred image enhancement method provided in this application embodiment;

[0031] Figure 2 The architecture of the blurred image enhancement model provided in the embodiments of this application;

[0032] Figure 3 A schematic diagram of the learning stage 1 of the blurred image enhancement model provided in the embodiments of this application;

[0033] Figure 4A schematic diagram of the learning stage 2 of the blurred image enhancement model provided in the embodiments of this application;

[0034] Figure 5 A schematic diagram of the learning stage 3 of the blurred image enhancement model provided in the embodiments of this application;

[0035] Figure 6 A schematic diagram of the hardware architecture of the electronic device provided in this application. Detailed Implementation

[0036] The technical solutions in the embodiments of this application will be clearly and thoroughly described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; the word "and / or" in the text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone.

[0037] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0038] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0039] The term "user interface (UI)" used in the following embodiments of this application refers to the medium interface through which an application or operating system interacts and exchanges information with the user. It realizes the conversion between the internal form of information and the form that the user can accept. The user interface is source code written in a specific computer language such as Java or Extensible Markup Language (XML). The interface source code is parsed and rendered on the electronic device, ultimately presenting content that the user can recognize. A common form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed graphically. It can be visible interface elements such as text, icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets displayed on the screen of an electronic device.

[0040] First, we will introduce several related concepts involved in the embodiments of this application.

[0041] Multi-resolution blurred images

[0042] In the field of image processing, a multi-resolution image refers to multiple images with different resolutions. In the embodiments of this application, for a blurred image, multiple blurred images with different resolutions can be generated by upsampling or downsampling. Based on multi-resolution blurred images, there is blur invariance, that is, the basic features of blurred images with different resolutions remain unchanged, all originating from the same clear image.

[0043] Fuzzy kernel

[0044] From a theoretical mathematical perspective, the blur kernel is equivalent to a function that describes how an image changes from a sharp state to a blurred state. Typically, this blurring process can be represented by a convolution operation; that is, convolving the sharp image with the blur kernel, and then adding noise, ultimately yields the blurred image. The image blurring process can be described by the following formulas 1 and 2. Formula 1 represents temporal convolution, where y... i Let I be the i-th blurred image, k be the blur kernel, and I be the image with the blurred image. i Let be the i-th clear image, n be the noise, 1≤i≤N, N≥2, and this application uses N=3 as an example. Formula 2 is expressed as a frequency domain multiplication, that is, Formula 1 is transformed from the time domain to the frequency domain after Fourier transform. F(y) is the blurred image, F(k) is the blur kernel, F(i) is the clear image, and F(n) is the noise.

[0045] Formula 1:

[0046] Formula 2: F(y)=F(k)×F(I)+F(n)

[0047] From a practical application perspective, the causes of image blurring are complex and varied. For example, camera shake, out-of-focus shots, defocusing, or movement of the subject can all lead to image blurring. The actual blurring process corresponding to each blurred image is unknown, making it difficult to accurately determine the blurring process. In other words, given a blurred image y... i At that time, due to the blur kernel k and the clear image I i Since both are unknown, it is difficult to estimate the accurate blur kernel k, and therefore difficult to restore the blurred image to a clear image I. i .

[0048] Blur image enhancement model

[0049] The blurred image enhancement model provided in this application is based on the principles of the aforementioned Formulas 1 and 2. This model can be based on the input blurred image y i Through calculation, a clear image I is output. i In other words, the calculation process of this model is to convert the time-domain convolution calculation into a frequency-domain multiplication calculation, which is equivalent to calculating the product of the blurred image F(y) and the inverse of the blur kernel F(k) to solve for the sharp image F(I).

[0050] loss function

[0051] In the learning process of a blurred image enhancement model, the loss function is used to measure the prediction accuracy of the model. Specifically, the loss function calculates the difference between the predicted result output by the blurred image enhancement model and the ideal result. By minimizing the loss function, the parameters used in the blurred image enhancement model are optimized in reverse, thereby learning a blurred image enhancement model that produces the closest result to the ideal.

[0052] To improve the quality of blurred images and enhance low-quality blurred images into high-quality sharp images, this application provides a method and electronic device for blurred image enhancement. The method includes: generating N blurred images of different resolutions based on the blurred image to be processed; inputting the N blurred images of different resolutions into an image enhancement model to calculate and output a sharp image. The image enhancement model is learned based on multiple sets of training images through the following stages: each set of training images includes one high-quality sharp image I and corresponding N low-quality blurred images y of different resolutions. iFor each training image, 1 ≤ i ≤ N: 1. Learn the Codebook based on the clear image I and the first loss function; 2. Learn the Transformer (i.e., the method of querying the Codebook) based on all training images and the second loss function; 3. Learn the Encoder, multi-resolution fusion module, and CFT based on all training images and the third loss function. This results in an image augmentation model containing an Encoder, Transformer, Codebook, Decoder, multi-resolution fusion module, and CFT.

[0053] In this model, at each stage of learning the blurred image enhancement model, a specific module is used as the learning objective, and a specific loss function is employed to constrain the learning process, thereby ensuring that the blurred image enhancement model can effectively learn the required characteristics. The learning objectives and loss functions used in each learning stage will be described in detail later and will not be elaborated upon here.

[0054] Implementing the blurry image enhancement method provided in the embodiments of this application or applying the electronic device provided in this application can bring the following beneficial effects:

[0055] (1) By utilizing the unchanged blur characteristics of multi-resolution images, a more accurate blur image enhancement model can be learned.

[0056] Specifically, learning a blurred image enhancement model involves solving for the inverse of the blur kernel, i.e., the inverse of k. Since, for the same sharp image I, its corresponding i blurred images y with different resolutions... i It possesses fuzziness invariance, meaning that fuzzy images y at different resolutions... i The basic characteristics remain unchanged, all originating from the same clear image I. Therefore, by establishing N sets of equations, i.e. This method solves for the inverse of k, compared to solving for it using a set of equations. To solve for the inverse of k, the equations are solved but the number of unknowns is almost not increased, thereby improving the accuracy of the inverse of the fuzz kernel, that is, the learned fuzzy image enhancement model is more accurate.

[0057] (2) Make full use of the loss function to constrain the learning process and learn a more accurate fuzzy image enhancement model.

[0058] Specifically, in the learning process of the enhanced image model, the characteristics of time-domain to frequency-domain transformation are utilized for calculation. Fourier transform is employed to convert the equations from a time-domain convolution representation to a frequency-domain multiplication representation. The problem is transformed into F(y) = F(k) × F(I) + F(n). Calculating the inverse of F(k) in the frequency domain can simplify the computation. In addition, not only is the time domain loss function used, but a frequency domain loss function is also added to optimize the inverse of F(k) in reverse, thereby further learning an accurate blurred image enhancement model.

[0059] refer to Figure 1 , Figure 1 An exemplary illustration shows the process flow of the blurry image enhancement method provided in this application.

[0060] like Figure 1 As shown, the method includes the following steps:

[0061] S101, Obtain the first image.

[0062] In one application scenario, each image captured by an electronic device can be enhanced using a default blurring method, and the resulting sharp image is then stored in a photo library. Therefore, in this scenario, the first image acquired by the electronic device can be any image captured by the user.

[0063] In another application scenario, a blurred image enhancement method is used to process the image actively selected by the user, and a clear image is output after processing. Therefore, in this scenario, the first image acquired by the electronic device can be any image selected by the user.

[0064] Optionally, the first image acquired by the electronic device may belong to one or more categories such as images of people, animals, or landscapes, and this application embodiment does not limit this.

[0065] S102, Detect whether the first image belongs to the blurry category.

[0066] Specifically, the electronic device can perform detection on all or only a local area of ​​the first image to determine whether the first image is a blurred image. Optionally, when the first image is a person image, only a local area with the face as the main subject can be extracted from the first image, and then this local area can be detected to determine whether the first image is a blurred image. When the detected blur level of the first image is greater than a preset value, the first image is considered to belong to the blurred image category, that is, the first image is regarded as the first blurred image.

[0067] This application does not impose specific limitations on the method used for fuzzy category detection. For example, the following methods can be used: edge detection, spectrum analysis, or by analyzing the statistical characteristics of the image, such as variance and entropy, to determine whether the image is fuzzy.

[0068] S103, after determining that the first image is a first blurred image, a first clear image is generated based on the first blurred image using a blurred image enhancement model.

[0069] Specifically, the electronic device has a pre-installed blur image enhancement model. After determining that the first image belongs to the blur category, the first image is used as the first blur image, and the blur image enhancement model is used to process the first blur image. Through the calculation of the blur image enhancement model, a first clear image is output. This first clear image is equivalent to improving the clarity based on the first blur image.

[0070] In one alternative implementation, since the features of different categories of images differ significantly, in order to ensure that a clear image can be accurately learned, the learning process of the blurred image enhancement model can be performed only on images of a specific category, thereby generating a blurred image enhancement model for that specific category. For example, the model can be learned only on the category of images of people. This application does not limit the specific image category.

[0071] In another possible approach, in order to broaden the application scope of the blurred image enhancement model, the model can be trained on multiple specific image categories during the learning process, thereby generating a blurred image enhancement model with good generalization ability. For example, it can be trained on multiple categories such as human images, animal images, and landscape images. This application embodiment does not limit the specific image categories.

[0072] For the architecture and learning process of this blurred image enhancement model, please refer to the following text. Figures 2-5 The details of this will not be elaborated here.

[0073] refer to Figure 2 , Figure 2 An example is shown illustrating the architecture of the blurred image enhancement model provided in this application.

[0074] like Figure 2 As shown, the blurred image enhancement model provided in this application is based on an encoder-decoder architecture, including: Encoder, SwinTransformer, Codebook, Decoder, Controllable Feature Transformation (CFT) module, multi-resolution fusion module, etc. The specific functions of each module are described below.

[0075] (1) The Encoder is used to encode the input image and compress it into a feature vector before outputting it. The Decoder is used to decode the input feature vector and output the feature vector as an image. The combination of Encoder and Decoder can be regarded as solving a mathematical problem.

[0076] This application does not impose any restrictions on the specific implementation of the Encoder and Decoder. Optionally, the Encoder and Decoder can be implemented using a Convolutional Neural Network (CNN), or alternatively, a Recurrent Neural Network (RNN), a Deep Neural Network (DNN), etc. Figure 2 This section will only use a residual neural network (ResNet) as an example. ResNet is a type of convolutional neural network that uses residual connections to address the vanishing and exploding gradient problems in deep networks.

[0077] This application addresses the multi-resolution blurred image y input to the Encoder. i There are no restrictions on the number or specific resolution. Figure 2 The example uses only three images with different resolutions. Specifically, it illustrates the acquisition of a multi-resolution blurred image y. i The method includes: after obtaining the blurred image y1, if the resolution of the blurred image y1 is N1×M1, then by downsampling the blurred image y1, a blurred image y2 with a resolution of N2×M2 and a blurred image y3 with a resolution of N3×M3 can be obtained.

[0078] This application does not impose specific restrictions on the residual blocks contained in the residual neural network. Figure 2 The residual neural network shown here is only illustrated with three residual blocks. Optionally, in addition to three residual blocks, there can be other residual blocks of two or more.

[0079] (2) SwinTransformer is a Transformer applied in the field of computer vision (CV). In this embodiment, SwinTransformer is mainly used to predict the index position of the corresponding high-quality feature sequence in the codebook based on the encoded blurred image data output by the encoder. The high-quality feature sequence in the codebook is then obtained based on the predicted index position in the codebook space. These high-quality feature sequences represent high-quality images. The predictive capability of SwinTransformer enables the model to index the closest high-quality image based on a low-quality blurred image.

[0080] In this embodiment, based on the learned codebook, a method for predicting high-quality image feature sequences corresponding to the codebook based on blurred images is further learned; that is, the Swing Transformer is further learned. For details on the learning method of the Swing Transformer, please refer to the following section. Figure 4 The details of this will not be elaborated here.

[0081] (3) A codebook is a technique for quantizing and encoding images. In the embodiments of this application, the codebook is mainly used to quantize and encode high-quality clear images into feature sequences. The codebook defines a large number of high-quality clear image feature sequences, and the blurred image is mapped onto the high-quality clear image feature sequences in the codebook, thereby realizing both feature extraction of image data and prediction of clear images.

[0082] In this embodiment, a codebook can be generated by learning from high-quality, clear images. The learning method for the codebook can be found below. Figure 3 The details of this will not be elaborated here.

[0083] (4) A controllable feature transformation (CFT) module is used to improve the performance of the blurred image enhancement model. In this embodiment, the CFT module controls the data processing flow between the Encoder and Decoder. Specifically, the CFT module employs a feature transformation layer with adjustable coefficients. When processing highly blurred images, it reduces features carrying the highly blurred image, i.e., reduces dependence on the input image, thereby improving the quality of the output image. When processing lightly blurred images, it increases features carrying the lightly blurred image, i.e., increases dependence on the input image, thereby preserving the true information of the input image. In this way, a balance can be struck between enhancing image quality and preserving the original image's realism, thereby improving the performance of the blurred image enhancement model.

[0084] In this embodiment, after learning the Codebook and Swin Transformer, the CFT module can be learned. For instructions on learning the CFT module, please refer to the following... Figure 5 The details of this will not be elaborated here.

[0085] (5) The multi-resolution fusion module is used to fuse i images of different resolutions into one image, which can effectively enhance the details and clarity of the image. Optionally, the multi-resolution fusion module can utilize a convolutional neural network (CNN), or it can also use a recurrent neural network (RNN), deep neural networks (DNN), etc. This invention uses a residual neural network (ResNet).

[0086] In this embodiment, after learning the Codebook and Swin Transformer, a multi-resolution fusion module can also be learned. For the learning method of the multi-resolution fusion module, please refer to the following... Figure 5 The details of this will not be elaborated here.

[0087] Figure 2 The diagram merely illustrates the basic architecture of one feasible blurred image enhancement model; alternative blurred image enhancement models may also include components that are more complex than those described above. Figure 2 The embodiments of this application do not limit the number of modules, such as the absence of CFT.

[0088] Next, combined Figures 3-5 This introduction describes the learning process of the blurred image enhancement model provided in this application. This learning process can consist of three stages, the first stage being... Figure 3 The Codebook and Decoder learning stages shown are shown; the second stage is... Figure 4 The SwinTransformer learning stage shown is the third stage of feature sequence prediction. Figure 5 The learning process of the Encoder, multi-resolution fusion module, and CFT module is shown. It can be seen that each learning stage has a specific learning objective, and a specific loss function is used to inversely optimize the learning objective, so that the final blurred image enhancement model predicts a sharp image that closely approximates the ideal sharp image.

[0089] refer to Figure 3 , Figure 3 The learning phase 1 of the blurred image enhancement model provided in this application is illustrated by way of example.

[0090] like Figure 3 As shown, this learning phase 1 is mainly based on learning the Codebook and Decoder using multi-resolution clear images.

[0091] Specifically, a first initial model is constructed, which includes an Encoder, a Codebook, and a Decoder. The first training data is used as input to the first initial model, and a first loss function is applied to constrain it, thereby learning the Codebook and Decoder.

[0092] The first training data includes multiple sets of high-quality clear images. Each set of high-quality clear images includes N high-quality clear images with different resolutions. These N high-quality clear images are obtained by downsampling the first clear image and then using N-1 clear images with different resolutions.

[0093] The first loss function includes: time-domain loss, frequency-domain loss, dictionary quantization loss, and generative adversarial network (GAN) loss.

[0094] The specific learning process for Codebook includes:

[0095] (1) Feature extraction: The Encoder extracts features from the multi-resolution image and uses the extracted features as input to the Codebook.

[0096] Specifically, a group of N high-quality clear images is input to the Encoder in batches. After each group of N high-quality clear images is input to the Encoder, the Encoder extracts N groups of feature vectors from the N high-quality clear images respectively, which is equivalent to encoding the N high-quality clear images to obtain the corresponding N groups of feature vectors. The Encoder specifically uses a residual network to implement encoding, and the residual network includes N residual blocks with residual connections, Figure 3 where N=3 is only taken as an example for illustration, N may be a value greater than or equal to 2, and the specific value of N is not limited in the embodiments of the present application.

[0097] (2) Codebook construction: further quantize the features extracted based on the Encoder, and construct a codebook containing a plurality of codewords.

[0098] Specifically, a group of N groups of feature vectors extracted from N high-quality clear images is input to the Codebook in batches. After each time the N groups of feature vectors are input to the Codebook, the Codebook uses M groups of feature vectors (usually M<N) to encode the input N groups of feature vectors, so as to form a series of codewords for representing images, and the codewords constitute the codebook.

[0099] (3) Backward optimization of Codebook and Decoder: a first loss function is used to optimize the Codebook and the Decoder. Specifically, the first loss function includes the functions shown in the following Formulas 3, 4, and 5:

[0100] Formula 3:

[0101] Formula 4:

[0102] Formula 5:

[0103] where Formula 3 is used to represent the codebook loss during the Codebook learning process. Z h is used to represent the output of the last residual block in the Encoder, Z c is used to represent the output of the Codebook.

[0104] where Formula 4 is used to represent the temporal loss and frequency domain loss of the multi-resolution image during the learning process of the Codebook and the Decoder. is used to represent the ideal multi-resolution clear image in the time domain, I i represents the multi-resolution image output by the first initial model in the time domain, that is, the multi-resolution image output by the residual block in the Decoder, F(I) is used to represent an ideal, sharp, multi-resolution image in the frequency domain. i ) represents the multi-resolution image output by the first initial model in the frequency domain, that is, the multi-resolution image output by the residual block in the Decoder.

[0105] Formula 5 represents the GAN loss during the Codebook and Decoder learning process. Where I... i The first clear image predicted by the Decoder output. It is an ideal multi-resolution clear image in the time domain.

[0106] In one feasible approach, the learning target in stage 1 also includes the Encoder. However, after stage 1 is completed, only the Encoder learned in stage 1 is used as an intermediate result. The Codebook and Decoder learned in stage 1 are used as the Codebook and Decoder in the final image augmentation model. Only the Encoder learned in the final stage 3 is used as the Encoder in the final image augmentation model.

[0107] refer to Figure 4 , Figure 4 The learning phase 2 of the blurred image enhancement model provided in this application is illustrated by way of example.

[0108] like Figure 4 As shown, this learning phase 2 is mainly based on learning the SwinTransformer using multi-resolution blurred image and sharp image pairs.

[0109] Specifically, a second initial model is constructed, which includes an Encoder, a SwingTransformer, a Codebook, and a Decoder. The second training data is used as input to the second initial model, and a second loss function is applied to constrain it, thereby learning the SwingTransformer. The Codebook and Decoder in this second initial model are trained in stage 1 and locked in stage 2.

[0110] The second training data includes multiple sets of low-quality blurred images and corresponding high-quality sharp images. Each set of low-quality blurred images specifically comprises N low-quality blurred images of different resolutions, which are obtained by downsampling the first blurred image into N-1 different-resolution blurred images. Similarly, each set of high-quality sharp images also corresponds to N high-quality sharp images of different resolutions, which are obtained by downsampling the first sharp image into N-1 different-resolution sharp images.

[0111] The second loss function includes: time domain loss, frequency domain loss, and index classification loss.

[0112] The specific learning process for SwinTransformer includes:

[0113] (1) Feature extraction: The Encoder extracts features from the multi-resolution image and uses the extracted features as input to the SwinTransformer.

[0114] Specifically, a set of N low-quality blurred images is input into the Encoder in batches. After each set of N low-quality blurred images is input into the Encoder, the Encoder extracts N sets of feature vectors from the N low-quality blurred images and then inputs the N sets of feature vectors into the SwingTransformer.

[0115] (2) Learn SwinTransformer: Use SwinTransformer to extract N sets of feature vectors based on the Encoder to predict the codeword sequences that are mapped in the Codebook. These codeword sequences represent the image representation in the codebook space.

[0116] Specifically, the Swin Transformer includes a hierarchical multi-scale architecture and a moving window-based self-attention mechanism to obtain the global composition and context of multi-resolution blurred images based on the feature vector output by the encoder. Following the self-attention block, a linear layer projects the features. Overall, the Swin Transformer encodes feature Z... l As input, predict Z in probabilistic form. l The corresponding codeword sequence. Then, the predicted codeword sequence retrieves the corresponding codeword from the learned codebook, thus forming the quantized feature Z. c Finally, a high-quality, sharp image is predicted using a fixed decoder.

[0117] (3) Backward optimization of SwinTransformer: A second loss function is used to optimize SwinTransformer. Since the parameters in SwinTransformer affect its indexing and classification capabilities, a second loss function is also needed. Through iterative optimization and evaluation, the parameters are adjusted to minimize the second loss function, thereby improving the performance of SwinTransformer and learning the final SwinTransformer.

[0118] Specifically, the second loss function includes the functions shown in Equations 6 and 7 below:

[0119] Formula 6:

[0120] in,

[0121] Formula 7:

[0122] Formula 6 represents the index classification loss during the learning process of SwinTransformer. For index loss, This results in content loss. l Z is the feature vector of the blurred image output by the encoder. c Quantization features in the codebook indexed by SwingTransformer.

[0123] Formula 7 is used to represent the temporal and frequency domain losses of multi-resolution images during the Codebook learning process. Used to represent an ideal, sharp, multi-resolution image in the time domain, I i This represents the multi-resolution image output by the first initial model in the time domain, i.e., the multi-resolution image output by the residual block in the Decoder. F(I) is used to represent an ideal, sharp, multi-resolution image in the frequency domain. i ) represents the multi-resolution image output by the first initial model in the frequency domain, that is, the multi-resolution image output by the residual block in the Decoder.

[0124] In one feasible approach, stage 2 also includes learning an encoder. However, after stage 2 is completed, only the encoder learned in stage 2 is used as an intermediate result. The SwinTransformer learned in stage 1 is used as the final SwinTransformer in the image augmentation model, and only the encoder learned in the final stage 3 is used as the final encoder in the image augmentation model. In other words, each stage of the image augmentation model needs to learn an encoder, and the encoders learned in stages 1 and 2 are only used as intermediate results.

[0125] refer to Figure 5 , Figure 5 The learning phase 3 of the blurred image enhancement model provided in this application is illustrated by way of example.

[0126] like Figure 5 As shown, this learning phase 3 mainly learns the Encoder, multi-resolution fusion module, CFT, etc., based on multi-resolution blurred image and clear image pairs.

[0127] Specifically, a third initial model is constructed, which includes an Encoder, a SwingTransformer, a Codebook, a Decoder, a multi-resolution fusion module, and a CFT. The third training data is used as input to the third initial model, and a third loss function is applied to constrain it, thereby learning the multi-resolution fusion module, CFT, etc. The Codebook and Decoder in this third initial model are trained in stage 1 and locked in stage 3, while the SwingTransformer in this third initial model is trained in stage 2 and locked in stage 3.

[0128] The third set of training data includes multiple sets of low-quality blurred images and corresponding high-quality sharp images. Each set of low-quality blurred images specifically comprises N low-quality blurred images with different resolutions, which are obtained by downsampling the first blurred image into N-1 different-resolution blurred images. Similarly, each set of high-quality sharp images also corresponds to N high-quality sharp images with different resolutions, which are obtained by downsampling the first sharp image into N-1 different-resolution sharp images.

[0129] The third loss function includes: time domain loss, frequency domain loss, and index classification loss.

[0130] The specific learning process regarding the Encoder, multi-resolution fusion module, and CFT includes:

[0131] (1) Feature extraction: The Encoder extracts features from the multi-resolution image and uses the extracted features as input to the SwinTransformer.

[0132] Specifically, a set of N low-quality blurred images is input into the Encoder in batches. After each set of N low-quality blurred images is input into the Encoder, the Encoder extracts N sets of feature vectors from the N low-quality blurred images, and then inputs the N sets of feature vectors into the SwinTransformer and the multi-resolution fusion module respectively.

[0133] (2) Learn Encoder, Multi-resolution Fusion Module, and CFT:

[0134] The multi-resolution fusion module extracts N sets of feature vectors based on the encoder. N multi-resolution fusion modules then fuse these N sets of feature vectors into N fused feature vectors, each corresponding to a resolution of one of the N blurred images. For example, with N=3, the three residual blocks in the encoder output three sets of feature vectors respectively. The first multi-resolution fusion module fuses these three sets of feature vectors into a first fused feature vector, i.e., ... The second multi-resolution fusion module fuses the three sets of feature vectors into a second fused feature vector, i.e. The third multi-resolution fusion module fuses the three sets of feature vectors into a third fused feature vector, i.e. The first, second, and third fusion feature vectors correspond to different resolutions. The resolution corresponding to the first fusion feature vector is the first resolution corresponding to the first blurred image to be processed, denoted as N1×M1. The resolution corresponding to the second fusion feature vector is the second resolution blurred image obtained after downsampling the first blurred image, denoted as N2×M2. The resolution corresponding to the third fusion feature vector is the third resolution blurred image obtained after downsampling the second blurred image, denoted as N3×M3.

[0135] The SwinTransformer is used to extract N sets of feature vectors based on the Encoder to predict the codeword sequences mapped to them in the Codebook. Then, the predicted codeword sequences are retrieved from the learned Codebook to form the quantized feature Z. c Finally, Z is obtained through three residual blocks in the Decoder. c Decoding is performed to obtain the corresponding features. For example, with N=3, the three residual blocks in the decoder output three sets of feature vectors respectively. The first residual block will decode Z... c Decoded into the corresponding first intermediate feature, denoted as The second residual block will be Z c Decoded into the corresponding second intermediate feature, denoted as The third residual block will be Z c Decoded into the corresponding third intermediate feature, denoted as

[0136] The CFT is used to adjust the N sets of intermediate feature vectors based on the N sets of fused feature vectors; wherein, one CFT is used to adjust one set of intermediate feature vectors based on one set of fused feature vectors. For example, taking N=3 as an example, the first CFT is based on... To adjust get Then this Input to the first residual block, the second CFT is based on To adjust get Then this Input to the second residual block, the third CFT based on To adjust get Then this The input is fed into the third residual block, resulting in the Decoder outputting three clear predicted images. For details on the CFT method of adjusting intermediate feature vectors based on fused feature vectors, please refer to [link to documentation]. Figure 5 The CFT structure shown is given, where α and β are affine transformation parameters, and w is the weight matrix.

[0137] (3) Reverse optimization of the Encoder, multi-resolution fusion module, and CFT: A third loss function is used to optimize the Encoder, multi-resolution fusion module, and CFT. Since the parameters in the Encoder, multi-resolution fusion module, and CFT affect the performance of the entire model in outputting a clear image, a third loss function is also needed. Through iterative optimization and evaluation, the parameters are adjusted to minimize the third loss function, thereby improving the performance of the entire module and learning the final model. The third loss function consists of the multi-resolution time-frequency loss and the GAN loss. For an introduction to this third function, please refer to the descriptions of Equations 4 and 5 above; they will not be elaborated upon here.

[0138] In one feasible approach, the learning objective in the final stage 3 also includes the Encoder, and the Encoder learned in stage 3 is used as the Encoder in the final image augmentation model.

[0139] In summary, the blurred image enhancement method provided in this application aims to learn a more accurate blurred image enhancement model. Specifically, the learning process of the blurred image enhancement model is described through the aforementioned three stages: based on a multi-resolution processing strategy, time-frequency loss is added as a constraint to achieve more accurate image deblurring.

[0140] The following section describes the hardware and software architecture and product form of the electronic device involved in this application.

[0141] Electronic devices can be equipped with Or other portable terminal devices with different operating systems, such as mobile phones, tablets, desktop computers, laptops, handheld computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, as well as cellular phones, personal digital assistants (PDAs), augmented reality (AR) devices, virtual reality (VR) devices, artificial intelligence (AI) devices, wearable devices, in-vehicle devices, smart home devices and / or smart city devices, etc.

[0142] Figure 6 A schematic diagram of the hardware architecture of an electronic device is shown.

[0143] like Figure 6 As shown, the electronic device may include: a processor 110, an external memory interface 120, an internal memory 121, a camera 130, a display screen 140, and a sensor module 160. The sensor module 160 may include a pressure sensor 160A, a touch sensor 160B, etc.

[0144] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device. In other embodiments of this application, the electronic device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware. For example, the electronic device may also include buttons, motors, indicators, and a subscriber identification module (SIM) card interface, etc. As another example, the sensor module may also include: a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a fingerprint sensor, a temperature sensor, a bone conduction sensor, etc.

[0145] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0146] The controller can serve as the nerve center and command center of an electronic device. Based on the instruction opcode and timing signals, the controller generates operation control signals to control the fetching and execution of instructions.

[0147] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0148] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0149] In this embodiment, the processor 110 implements the learning of a blurred image enhancement model, and can also be used to deblur blurred images using the learned blurred image enhancement model to output a clear image. Optionally, the blurred image enhancement model can be pre-learned by other devices and then stored in other electronic devices, enabling other electronic devices to enhance blurred images using this blurred image enhancement mode. For the specific implementation process of the processor 110 learning the blurred image enhancement model and enhancing blurred images, please refer to the preceding text. Figures 1-5 The details of this will not be elaborated here.

[0150] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a limitation on the structure of the electronic device. In other embodiments of this application, the electronic device may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0151] Internal memory 121 may include one or more random access memory (RAM) and one or more non-volatile memory (NVM).

[0152] Random access memory can include static random-access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), and double data rate synchronous dynamic random access memory (DDR SDRAM, such as fifth-generation DDR SDRAM, which is generally called DDR5 SDRAM). Non-volatile memory can include disk storage devices and flash memory.

[0153] Random access memory (RAM) can be directly read and written by the processor 110. It can be used to store executable programs (such as machine instructions) of the operating system or other running programs, as well as user and application data. Non-volatile memory can also store executable programs and user and application data, and can be pre-loaded into RAM for direct read and write by the processor 110.

[0154] The external memory interface 120 can be used to connect to external non-volatile memory, thereby expanding the storage capacity of the electronic device. The external non-volatile memory communicates with the processor 110 through the external memory interface 120 to perform data storage functions. For example, music, video, and other files can be stored in the external non-volatile memory.

[0155] In this embodiment of the application, the aforementioned memory can be used to store blurred image enhancement models, etc.

[0156] Electronic devices implement display functions through a GPU, a display screen 140, and an application processor. The GPU is a microprocessor for image processing, connecting the display screen 140 and the application processor. The GPU performs mathematical and geometric calculations and is used for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0157] Display screen 140 is used to display images, videos, etc. Display screen 140 includes a display panel. The display panel can be a liquid crystal display (LCD). The display panel can also be manufactured using organic light-emitting diodes (OLEDs), active-matrix organic light-emitting diodes (AMOLEDs), flexible light-emitting diodes (FLEDs), miniled, microLEDs, micro-OLEDs, quantum dot light-emitting diodes (QLEDs), etc. In some embodiments, the electronic device may include one or N displays 140, where N is a positive integer greater than 1.

[0158] In this embodiment, the display screen 140 can be used to display the interface provided by the camera application and the interface provided by the gallery, specifically including but not limited to the preview interface, shooting interface, settings interface, etc. The camera application may display controls / icons / options that allow the user to adjust the zoom level and image size, etc.

[0159] Electronic devices can achieve shooting functions through ISP, camera 130, video codec, GPU, display 140 and application processor.

[0160] The ISP is used to process data fed back from the camera 130. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, converting it into an image visible to the naked eye. The ISP can also perform algorithmic optimization on image noise and brightness. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 130.

[0161] Camera 130 is used to capture still images or videos. Camera 130 includes the image sensor described above. An object is projected onto a photosensitive element through a lens to generate an optical image. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the electronic device may include one or N cameras 130, where N is a positive integer greater than 1.

[0162] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when an electronic device is selecting a frequency, a DSP can perform a Fourier transform on the frequency energy.

[0163] Video codecs are used to compress or decompress digital video. Electronic devices can support one or more video codecs. This allows the electronic device to play or record video in various encoded formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0164] An NPU (Neural Processing Unit) is a computational processor for neural networks (NNs). By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0165] In the embodiments of this application, the neural network can be used to perform domain transformation on the image during the image processing stage, or to perform image magnification, cropping, and other processing.

[0166] Pressure sensor 160A is used to sense pressure signals and convert them into electrical signals. In some embodiments, pressure sensor 160A can be disposed on display screen 140. There are many types of pressure sensors 160A, such as resistive pressure sensors, inductive pressure sensors, and capacitive pressure sensors. A capacitive pressure sensor may include at least two parallel plates with conductive material. When force is applied to pressure sensor 160A, the capacitance between the electrodes changes. The electronic device determines the pressure intensity based on the change in capacitance. When a touch operation is applied to display screen 140, the electronic device detects the intensity of the touch operation based on pressure sensor 160A. The electronic device can also calculate the touch position based on the detection signal from pressure sensor 160A. In some embodiments, touch operations applied to the same touch position but with different touch operation intensities can correspond to different operation commands. For example, when a touch operation with an intensity less than a first pressure threshold is applied to the SMS application icon, a command to view an SMS is executed. When a touch operation with an intensity greater than or equal to the first pressure threshold is applied to the SMS application icon, a command to create a new SMS is executed.

[0167] Touch sensor 160B, also known as a "touch panel," can be located on display screen 140. The touch sensor 160B and display screen 140 together form a touchscreen, also known as a "touchscreen." Touch sensor 160B is used to detect touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 140. In other embodiments, touch sensor 160B may also be located on the surface of the electronic device, in a different position than display screen 140.

[0168] It should be understood that each step in the above method embodiments can be completed by integrated logic circuits in the processor hardware or by instructions in software form. The method steps disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or being executed by a combination of hardware and software modules in the processor.

[0169] This application also provides an electronic device that may include a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method performed by the electronic device as described in any of the above embodiments.

[0170] This application also provides a chip system including a processing circuit and an interface circuit. The interface circuit is used to receive computer instructions and transmit them to the processing circuit. The processing circuit is used to execute the computer instructions to implement the method performed by the electronic device as in any of the above embodiments.

[0171] This application also provides a chip system including at least one processor for implementing the methods executed by the electronic device in any of the above embodiments. In one possible design, the chip system further includes a memory for storing program instructions and data, the memory being located within or outside the processor.

[0172] A chip system can consist of chips or include chips and other discrete components.

[0173] Optionally, there may be one or more processors in the chip system. The processor can be implemented in hardware or software. When implemented in hardware, the processor can be a logic circuit, integrated circuit, etc. When implemented in software, the processor can be a general-purpose processor, implemented by reading software code stored in memory.

[0174] Optionally, the chip system may contain one or more memories. These memories may be integrated with the processor or disposed separately; this application does not limit this. For example, the memory may be a non-transient processor, such as a read-only memory (ROM), which may be integrated with the processor on the same chip or disposed on different chips. This application does not specifically limit the type of memory or the arrangement of the memory and processor.

[0175] For example, the chip system may be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a micro controller unit (MCU), a programmable logic device (PLD), or other integrated chips.

[0176] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method executed by the electronic device in any of the above embodiments.

[0177] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the method executed by the electronic device as described in any of the above embodiments.

[0178] The various embodiments of this application can be combined arbitrarily to achieve different technical effects.

[0179] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0180] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method for enhancing a blurred image, characterized in that, The method includes: Obtain N blurred images, wherein the N blurred images have different resolutions and the N blurred images have the same blur features; The N blurred images are used as input data for the image enhancement model, which includes an encoder, a converter, a codebook, a decoder, N multi-resolution fusion modules, and N controllable feature transformation (CFT) modules. The encoder encodes the N blurred images into N corresponding feature vectors. The converter indexes the N sets of feature vectors and maps them to the N sets of codewords in the codebook. The decoder outputs N sets of intermediate feature vectors based on the indexed N sets of codewords. The N sets of feature vectors are fused into N sets of fused feature vectors by the N multi-resolution fusion modules respectively, and the N sets of fused feature vectors correspond to the resolution of the N blurred images; The N CFTs are used to adjust the N sets of intermediate feature vectors based on the N sets of fused feature vectors; wherein, 1 CFT is used to adjust 1 set of intermediate feature vectors based on 1 set of fused feature vectors. The decoder outputs the corresponding first clear image based on the adjusted N sets of intermediate feature vectors. The image enhancement model is learned based on multiple sets of training images and a loss function. Each set of training images includes: N training blurred images with the same resolution as the N blurred images and corresponding training clear images; the loss function includes a time-domain loss function and a frequency-domain loss function.

2. The method according to claim 1, characterized in that, The loss function also includes an index classification loss function; The encoder and the N CFTs are learned based on the blurred and sharp images in the training images, as well as the temporal loss function, the frequency domain loss function, and the index classification loss function.

3. The method according to claim 1, characterized in that, The loss function also includes the codebook loss function and the GAN (Generative Adversarial Network) loss function. The codebook is learned based on clear images from the training images, as well as the temporal loss function, the frequency domain loss function, the codebook loss function, and the GAN loss function.

4. The method according to claim 1, characterized in that, The loss function also includes an index classification loss function; The converter is learned based on the blurred and sharp images in the training images, as well as the temporal loss function, the frequency domain loss function, and the index classification loss function.

5. The method according to claim 1, characterized in that, The encoder includes N residual blocks, and the decoder includes N residual blocks; The encoder's N residual blocks are used to encode the N blurred images into corresponding N sets of feature vectors; The first residual block of the decoder is used to receive the output of other residual blocks and output the first clear image.

6. The method according to claim 1, characterized in that, Obtaining N blurred images specifically includes: Acquire a first blurred image, wherein the first blurred image is at a first resolution; The first blurred image is downsampled to obtain N-1 blurred images, the N-1 blurred images have different resolutions, and the resolution of the N-1 blurred images is less than the first resolution.

7. The method according to claim 6, characterized in that, The first blurred image is one or more of the following types of images: images of people, images of animals, and images of landscapes.

8. The method according to claim 6, characterized in that, The blurriness of the first blurred image is greater than a preset value.

9. The method according to claim 6, characterized in that, Acquiring the first blurred image specifically includes: capturing the first blurred image using a camera.

10. The method according to claim 1, characterized in that, N=3。 11. An electronic device, characterized in that, It includes one or more memories, one or more processors, and a computer program stored on the memories, the processor executing the computer program to implement the steps of the method according to any one of claims 1-10.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it performs the steps of the method according to any one of claims 1-10.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it performs the steps of the method according to any one of claims 1-10.

Images