An image restoration method, apparatus, device and medium

Abstract

This application discloses an image restoration method, apparatus, device, and medium. The method includes extracting multiple first image patches from a degraded image to be trained, extracting a first Hankel matrix corresponding to each of the multiple first image patches, folding each first Hankel matrix into corresponding multiple first tensors at different scales, training a generative diffusion model based on the multiple first tensors, and recovering the target image from the multiple second tensors of the target degraded image using the trained generative diffusion model. This method, by constructing multi-scale tensor data and training the generative diffusion model at different scales, avoids local optima when using the generative diffusion model for degraded image restoration, thereby improving the accuracy of degraded image restoration.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image restoration method, apparatus, device and medium. Background Technology

[0002] Image restoration is one of the core research directions in computer vision and digital image processing, aiming to recover high-quality, clear images from degraded observation images (such as those with noise pollution, blurriness, or missing pixels).

[0003] In recent years, with the rapid development of deep learning technology, image restoration methods can be mainly divided into the following categories: 1) Deep learning methods based on supervised learning, such as convolutional neural networks, residual networks, U-Net, etc.; 2) Unsupervised methods based on generative models, such as generative adversarial networks, variational autoencoders, and diffusion models; 3) Traditional methods based on low-rank matrix optimization, which are suitable for small sample or high-noise scenarios.

[0004] Although existing methods have made some progress in the field of image restoration, there are some problems. For example, existing methods usually process at a fixed scale, which does not make sufficient use of multi-scale information, resulting in poor detail restoration. Summary of the Invention

[0005] This application aims to at least address the technical problems existing in the prior art. To this end, this application proposes an image restoration method, apparatus, device, and medium that can improve the accuracy of degraded image restoration.

[0006] In a first aspect, embodiments of this application provide an image restoration method, the method comprising:

[0007] Obtain the degraded image to be trained;

[0008] Extract multiple first image patches from the degraded image to be trained, and extract the first Hankel matrix corresponding to each of the multiple first image patches. Fold each first Hankel matrix into multiple corresponding first tensors according to different scales.

[0009] Each of the first Hankel matrix and the corresponding multiple first tensors are input into a preset generative diffusion model, so that the generative diffusion model learns the prior distribution of the image using stochastic differential equations, and obtains the trained generative diffusion model based on the prior distribution of the image.

[0010] In response to the recovery signal of the target degraded image, the target degraded image is acquired;

[0011] Extract multiple second image patches from the target degraded image, and extract the second Hankel matrix corresponding to each second image patch. Fold each second Hankel matrix into multiple corresponding second tensors according to different scales.

[0012] The target image is recovered from the various second tensors corresponding to each second Hankel matrix according to the generative diffusion model.

[0013] The embodiments of this application employ the above-described technical solution, which has at least the following beneficial effects: Existing methods typically process images at a fixed scale, resulting in insufficient detail restoration; This embodiment constructs multi-scale tensor data and trains a generative diffusion model at different scales, thereby avoiding local optima and improving the accuracy of degraded image restoration when using the generative diffusion model for image restoration.

[0014] In some embodiments of this application, recovering the target image from the plurality of second tensors corresponding to each second Hankel matrix according to the generative diffusion model includes:

[0015] Perform M iterations on the various second tensors corresponding to the second Hankel matrix to obtain the third image patch after the Mth iteration; where M is an integer greater than or equal to 2;

[0016] The target image is generated based on the third image block;

[0017] The process of the i-th iteration includes:

[0018] Determine the multiple tensors of the current input; when i is 1, the multiple tensors of the current input are the multiple second tensors; and when i is any number from 2 to M, the multiple tensors of the current input are multiple tensors obtained by first performing Hankel matrix transformation on the third image patch after the (i-1)th iteration, and then folding the transformed Hankel matrix according to the different scales.

[0019] Using the inverse equation corresponding to the stochastic differential equation, multiple candidate recovery samples that conform to the prior distribution of the image are sampled from the various tensors of the current input;

[0020] The multiple candidate recovery samples are expanded into multiple corresponding Hankel matrices;

[0021] Based on the expanded Hankel matrices, the third image patch is generated after the i-th iteration.

[0022] In some embodiments of this application, generating the third image patch after the i-th iteration based on the expanded multiple Hankel matrices includes:

[0023] The weighted mean of multiple expanded Hankel matrices is used to obtain the weighted mean Hankel matrix.

[0024] Based on the weighted average Hankel matrix, the third image patch is generated after the i-th iteration.

[0025] In some embodiments of this application, after sampling multiple candidate recovery samples conforming to the prior image distribution from the current multiple tensors using the inverse equation corresponding to the stochastic differential equation, the method further includes:

[0026] The multiple candidate recovery samples are corrected to obtain corrected multiple candidate recovery samples;

[0027] Expanding the multiple candidate recovery samples into corresponding multiple Hankel matrices includes:

[0028] The corrected candidate recovery samples are expanded into corresponding Hankel matrices.

[0029] In some embodiments of this application, the step of generating the third image patch after the i-th iteration based on the weighted average Hankel matrix includes:

[0030] The weighted mean Hankel matrix is ​​optimized by performing low-rank optimization using the alternating direction multiplier algorithm to obtain the low-rank optimized Hankel matrix.

[0031] Based on the low-rank optimized Hankel matrix, the third image patch is generated after the i-th iteration.

[0032] In some embodiments of this application, after generating the third image patch after the i-th iteration based on the Hankel matrix optimized by the low rank, the method further includes:

[0033] DC consistency correction is performed on the third image block after the i-th iteration.

[0034] In some embodiments of this application, the generation diffusion model is a denoising score matching loss function.

[0035] Secondly, embodiments of this application provide an image restoration apparatus, the apparatus comprising:

[0036] The image acquisition module is used to acquire the degraded image to be trained;

[0037] The first tensor generation module is used to extract multiple first image patches from the degraded image to be trained, and extract the first Hankel matrix corresponding to each of the multiple first image patches, and fold each first Hankel matrix into multiple corresponding first tensors according to different scales.

[0038] The model training module is used to input the various first tensors corresponding to each first Hankel matrix into a preset generation-diffusion model, so that the generation-diffusion model learns the prior distribution of the image using stochastic differential equations, and obtains the trained generation-diffusion model based on the prior distribution of the image.

[0039] A degraded image acquisition module is used to acquire the target degraded image in response to a recovery signal of the target degraded image;

[0040] The second tensor generation module is used to extract multiple second image blocks of the target degraded image, extract the second Hankel matrix corresponding to each second image block, and fold each second Hankel matrix into multiple corresponding second tensors according to different scales.

[0041] The target image recovery module is used to recover the target image from the multiple second tensors corresponding to each second Hankel matrix according to the generation diffusion model.

[0042] Thirdly, embodiments of this application provide an electronic device, including at least one control processor and a memory for communicatively connecting to the at least one control processor; the memory stores instructions executable by the at least one control processor, which, when executed by the at least one control processor, enable the at least one control processor to perform the aforementioned image restoration method.

[0043] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions for causing a computer to perform the aforementioned image restoration method.

[0044] It is understood that the beneficial effects of the second to fourth aspects compared with the related technologies are the same as the beneficial effects of the first aspect compared with the related technologies. Please refer to the relevant description in the first aspect above, which will not be repeated here. Attached Figure Description

[0045] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0046] Figure 1 This is a schematic flowchart of an embodiment of an image restoration method provided in this application;

[0047] Figure 2 This is a flowchart illustrating the training of a diffusion model according to one embodiment of this application;

[0048] Figure 3This is a schematic diagram of a process for restoring degraded images using a generative diffusion model, provided in one embodiment of this application.

[0049] Figure 4 This is a schematic diagram of the structure of an image restoration device provided in one embodiment of this application;

[0050] Figure 5 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this application. Detailed Implementation

[0051] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0052] In the description of this application, the use of terms such as "first," "second," etc., is for the purpose of distinguishing technical features only and should not be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated or the order of the technical features indicated.

[0053] In the description of this application, it should be understood that the orientation descriptions, such as up, down, etc., are based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application.

[0054] like Figure 1 As shown in one embodiment of this application, an image restoration method is provided, the method including steps S100 to S600:

[0055] Step S100: Obtain the degraded image to be trained.

[0056] Step S200: Extract multiple first image patches from the degraded image to be trained, and extract multiple first Hankel matrices corresponding to the multiple first image patches. Fold each first Hankel matrix into multiple corresponding first tensors according to different scales.

[0057] Step S300: Input the various first tensors corresponding to each first Hankel matrix into the preset generation-diffusion model, so that the generation-diffusion model learns the prior distribution of the image using stochastic differential equations, and obtains the trained generation-diffusion model based on the prior distribution of the image.

[0058] Step S400: In response to the recovery signal of the target degraded image, acquire the target degraded image.

[0059] Step S500: Extract multiple second image patches from the target degraded image, and extract multiple second Hankel matrices corresponding to the multiple second image patches. Fold each second Hankel matrix into multiple corresponding second tensors according to different scales.

[0060] Step S600: Recover the target image from the multiple second tensors corresponding to each second Hankel matrix according to the generation diffusion model.

[0061] In this embodiment, steps S100 to S300 are the training process for generating the diffusion model;

[0062] First, the degraded image to be trained is the degraded image used to train the generative diffusion model. Second, multiple image patches can be extracted from the degraded image to be trained using a sliding window. Then, a first Hankel matrix corresponding to each image patch can be generated through the sliding window. A Hankel matrix is ​​a special matrix structure in which all elements on each secondary diagonal (antidiagonal) are equal. Finally, the first Hankel matrix can be folded into various corresponding first tensors according to a preset scale. For example, the first Hankel matrix can be folded into tensors of three different scales: 192×192×16, 96×96×64, and 64×64×144.

[0063] It is important to note that multiple first Hankel matrices are folded into corresponding first tensors at the same scale; for example, there are... The first Hankel matrix, then The first Hankel matrix is ​​folded into: 192×192×16× 96×96×64× 64×64×144× Tensors of three different scales.

[0064] In subsequent steps, the generative diffusion model can be a score-based generative diffusion model (e.g., NCSN++). A diffusion model is a generative artificial intelligence technique designed based on a physical diffusion process. It is a network model that generates data by progressively adding and removing noise.

[0065] By inputting tensors of different scales into independent generative diffusion models, the models can learn the prior distribution of images at each scale. For example, the models can learn the gradient of the data distribution through stochastic differential equations (SDEs).

[0066] In some embodiments, the generative diffusion model is trained using a denoised score-matching loss.

[0067] In steps S400 to S600, the target degraded image is the degraded image that needs to be restored. The process of step S500 is similar to that of step S200, and will not be repeated here. Finally, the target image can be recovered from the various second tensors corresponding to each second Hankel matrix based on the trained generative diffusion model.

[0068] The image restoration method described in this embodiment has at least the following beneficial effects:

[0069] Existing methods typically process images at a fixed scale, resulting in insufficient detail recovery. This embodiment constructs a diffusion model using multi-scale tensor data and trains it at different scales to avoid local optima and improve the accuracy of degraded image restoration.

[0070] In some embodiments of this application, step S600, which involves recovering the target image from multiple second tensors corresponding to each second Hankel matrix based on a generative diffusion model, includes the following steps S610 to S620:

[0071] Step S610: Perform M iterations on the various second tensors corresponding to the second Hankel matrix to obtain the third image patch after the Mth iteration; where M is an integer greater than or equal to 2.

[0072] Step S620: Generate the target image based on the third image block;

[0073] The process of the i-th iteration includes the following steps S6210 to S6240:

[0074] Step S6210: Determine the multiple tensors of the current input; when i is 1, the multiple tensors of the current input are multiple second tensors; and when i is any number from 2 to M, the multiple tensors of the current input are multiple tensors obtained by first performing Hankel matrix transformation on the third image patch after the (i-1)th iteration, and then folding the transformed Hankel matrix according to different scales.

[0075] Step S6220: Using the inverse equation corresponding to the stochastic differential equation, multiple candidate recovery samples that conform to the prior distribution of the image are sampled from the multiple tensors of the current input.

[0076] Step S6230: Expand multiple candidate recovery samples into multiple corresponding Hankel matrices;

[0077] Step S6240: Generate the third image patch after the i-th iteration based on the expanded multiple Hankel matrices.

[0078] In this embodiment, step S600 performs M iterations until the final third image block is generated, which can be directly reconstructed into the target image.

[0079] like Figure 2 Here, we take the first and second iterations as examples. Let the number of image patches be 25, and let each image patch be divided into 3 types of tensors (192×192×16, 96×96×64, 64×64×144), that is:

[0080] First, the input data consists of three tensors: 192×192×16×25, 96×96×64×25, and 64×64×144×25.

[0081] Then, in the PC sampler, the inverse equation corresponding to the stochastic differential equation (SDE) is used to sample from the generative diffusion model to generate candidate solutions (i.e., candidate recovery samples) that conform to the prior distribution of the image. It should be noted that the PC sampler includes a predictor and a corrector, and only the function of the predictor is applied here.

[0082] Secondly, multiple candidate recovery samples are expanded into multiple corresponding Hankel matrices. For example, three candidate recovery samples are selected for three types of tensors and converted into three Hankel matrices.

[0083] Finally, based on the expanded three Hankel matrices, the third image patch after the first iteration is generated. For example, the Hankel matrix is ​​converted into an image patch using a technique based on the Hankel matrix.

[0084] Second iteration process:

[0085] First, the image patch obtained in the first iteration is transformed into a Hankel matrix, and then the Hankel matrix is ​​converted into a tensor.

[0086] Then, the tensor is used as the input to the PC sampler. The subsequent steps are similar to those of the first iteration, and will not be repeated here.

[0087] After M iterations, the final third image patch is obtained.

[0088] The method provided in this embodiment implements a priori process using a PC sampler, and then uses a stochastic differential equation (SDE) inverse process to gradually denoise during the iteration, which can generate more natural and more detailed recovery results.

[0089] In some embodiments of this application, step S6240, which generates the third image patch after the i-th iteration based on the expanded multiple Hankel matrices, includes the following steps S6241 to S6242:

[0090] Step S6241: The weighted mean of the expanded Hankel matrices is calculated to obtain the weighted mean Hankel matrix.

[0091] Step S6242: Based on the weighted average Hankel matrix, generate the third image patch after the i-th iteration.

[0092] In this embodiment, refer to Figure 3 (Let A, B, and C represent tensors of different scales, respectively.) Because the three tensors in the PC sampler are folded from the same second Hankel matrix, after passing through the PC sampler, they need to be converted back to Hankel matrices, resulting in three different Hankel matrices. Therefore, a weighted average of the three Hankel matrices is taken to convert them into a single Hankel matrix, which serves as the input for subsequent processes.

[0093] In some embodiments of this application, after sampling multiple candidate recovery samples that conform to the prior distribution of the image from the current multiple tensors using the inverse equation corresponding to the stochastic differential equation in step S6220, the method further includes step S6250:

[0094] Step S6250: Correct multiple candidate recovery samples to obtain corrected multiple candidate recovery samples;

[0095] Step S6230, which involves expanding multiple candidate recovery samples into corresponding Hankel matrices, includes the following step S6231:

[0096] Step S6231: Expand the corrected candidate recovery samples into corresponding Hankel matrices.

[0097] In this embodiment, a corrector is added to the PC sampler, which can correct the results obtained by the predictor using Langevin dynamics, making them more accurately match the marginal distribution of the current time step.

[0098] In some embodiments of this application, step S6242, which involves generating the third image patch after the i-th iteration based on the weighted average Hankel matrix, includes:

[0099] Step S6242a: Perform low-rank optimization on the weighted mean Hankel matrix according to the alternating direction multiplier algorithm to obtain the low-rank optimized Hankel matrix;

[0100] Step S6242b: Based on the Hankel matrix after low-rank optimization, the third image patch after the i-th iteration is generated.

[0101] In this embodiment, a low-rank constraint of the Alternating Direction Method of Multipliers (ADMM) is added, and the low-rank Hankel matrix is ​​optimized to capture the global structure of the image (such as edges and textures) by utilizing its redundancy characteristics, thus avoiding the generation of blur or distortion.

[0102] In some embodiments of this application, after generating the third image block after the i-th iteration based on the Hankel matrix optimized by low rank, the method further includes: step S6242c, performing DC consistency correction on the third image block after the i-th iteration.

[0103] In this embodiment, a DC step is added to ensure that the repaired third image block is consistent with the original observation at the known pixel position, thereby improving the stability of the recovered image.

[0104] For ease of understanding, such as Figure 2 and Figure 3 This paper provides an image restoration method, which includes the following steps:

[0105] Step S910, processing of training data; such as Figure 2 The steps include S911 to S913.

[0106] Step S911: Obtain the degraded image to be trained. (RGB three channels);

[0107] Step S912, the degraded image to be trained Blocking and filling;

[0108] For degraded images Perform edge fill (e.g., fill from 256×256 to 320×320) to eliminate boundary effects;

[0109] A sliding window is used to extract image patches (e.g., the size of the image patch is 64×64×3), generating a total of... The first image patch of a local area ;

[0110] Step S913: Construct the first Hankel matrix;

[0111] For each first image block via sliding window Generate the first Hankel matrix:

[0112] ;

[0113] By setting an appropriate window size It can control the redundancy and low rank of the matrix.

[0114] Step S914: Generate multiple first tensors of different scales;

[0115] Will The first Hankel matrix is ​​folded into three tensors according to different scales:

[0116] Scale 1: 192×192×16×k;

[0117] Scale 2: 96×96×64×k;

[0118] Scale 3: 64×64×144×k;

[0119] Step S920: Generate training for the diffusion model;

[0120] like Figure 2 We chose to use a score-based generative model (NCSN++).

[0121] Learn the gradient of the data distribution using stochastic differential equations (SDEs).

[0122] The first tensors at three different scales are input into independent generative diffusion models to learn the prior distribution of images at different scales. The loss function used here is the denoising score matching (DSM) loss.

[0123] Step S930 involves restoring the degraded target image based on a generative diffusion model; this includes steps S931 to S935.

[0124] The core of the entire iterative restoration process is solving a conditional generation problem, the goal of which is to recover the degraded image from the target image. A clear image was recovered from the image. Its optimization objective can be expressed as: given the observed values Under the condition of maximizing the posterior probability Using Bayes' theorem, this can be transformed into:

[0125] ;

[0126] in:

[0127] It is a data consistency item that ensures the recovered image matches the observed values;

[0128] It is a priori term that guides the restoration result to conform to the statistical distribution of natural images;

[0129] Step S931, Initialization;

[0130] Degrade the target image The second Hankel matrix As the initial input.

[0131] It should be noted that the target image here is degraded. The process of generating the second Hankel matrix is ​​similar to the degenerate image described above. The process of generating the first Hankel matrix is ​​similar and will not be described in detail here.

[0132] Here It is a degraded image of the target. of The second Hankel matrix of local image patches The aggregation result.

[0133] Step S932: Construct multi-scale tensors;

[0134] In the Hankel matrix Based on this, three second tensors with sizes of 192×192×16×k, 96×96×64×k, and 64×64×144×k are constructed as inputs to the subsequent network.

[0135] Step S933, alternate optimization;

[0136] The entire alternating optimization process is shown below, and its core is:

[0137] 1) PC (Predictor-Corrector) sampler (prior); it is the process of using the generative diffusion model that has been trained above.

[0138] 2) ADMM (Alternating Direction Method of Multipliers) (Low-rank constraint);

[0139] 3) DC (Data Consistency);

[0140] The core process is as follows:

[0141] First, candidate recovery samples are generated using a pre-trained generative diffusion model. ;

[0142] Then, solve the low-rank constraint problem;

[0143] Finally, the reconstruction results are enhanced in detail to preserve high-frequency information;

[0144] The process involves a three-step iterative loop; described in detail below:

[0145] Step S9331, the PC Sampler is a numerical method for solving the inverse SDE. Its function is to sample from the pre-trained score generation diffusion model to generate a model that conforms to the prior distribution. The candidate solution. It consists of two steps:

[0146] Predictor: Uses numerical methods such as the Euler-Maria method to directly predict the next sample based on the inverse SDE formula. It is responsible for "advancing" the sampling process.

[0147] ;

[0148] , : Tensors of the current and previous time steps (in the Hankel domain);

[0149] : No. The noise scale of the step;

[0150] Pre-trained score network, approximation That is, the prior gradient;

[0151] Random Gaussian noise.

[0152] Corrector: Uses Langevin dynamics to correct the results obtained by the predictor, making them more accurately match the marginal distribution of the current time step. It is responsible for "refining" and "correcting";

[0153] ;

[0154] The number of internal iterations in Langevin dynamics;

[0155] Step length.

[0156] Step S9332: Multi-scale fusion and averaging;

[0157] like Figure 3 Because the three tensors in the PC sampler are folded from the same second Hankel matrix, and each tensor needs to be converted back to a Hankel matrix after passing through the PC sampler, resulting in three different Hankel matrices, a weighted average of the three Hankel matrices is taken to convert them into a single Hankel matrix, which serves as the input for subsequent processes.

[0158] Step S9333: ADMM is used to solve the low-rank constrained optimization problem of the Hankel matrix. After being updated by the PC sampler, the tensor is expanded back to its original Hankel matrix form. The low-rank optimization problem takes the form of: ;

[0159] LMaFit (a low-rank matrix factorization algorithm without SVD) is used to efficiently solve this problem, and its update formula is:

[0160] ;

[0161] ;

[0162] ;

[0163] ; V: The two factor matrices of the low-rank matrix factorization. Augmented Lagrange multipliers Penalty parameters Conjugate transpose : Operations to expand a tensor back into a Hankel matrix;

[0164] After ADMM update, the low-rank Hankel matrix It requires the inverse operator. Convert back to third image patch .

[0165] Step S9334, DC consistency correction;

[0166] The DC step ensures that the repaired third image patch maintains consistency with the original observations at known pixel locations. The optimization problem is:

[0167] ;

[0168] : Downsampling matrix (here, a mask matrix representing the location of missing pixels) Observed degraded image patches Image patch estimation obtained from the ADMM step, Balance parameters;

[0169] The termination condition for step S933 is when the number of iterations reaches a certain threshold. Stop at time, among which The default value is 1000.

[0170] Finally, the target image is reconstructed based on the generated third image patches.

[0171] The image restoration method provided in this application has at least the following beneficial effects:

[0172] (1) Traditional supervised learning methods rely on a large number of high-quality reference images, but it is difficult to obtain accurately paired degraded-clear images in real-world scenarios. This embodiment adopts an unsupervised generative diffusion model, which only requires a single degraded image to learn the data distribution, thus reducing data requirements.

[0173] (2) Existing generative adversarial networks (GANs) are prone to training instability, pattern collapse and artifact problems. This embodiment introduces a score-based generative diffusion model, which gradually denoises through the stochastic differential equation (SDE) reverse process to generate more natural and detailed recovery results.

[0174] (3) Traditional low-rank methods rely on manually designed linear constraints, which are difficult to model the nonlocal structure of complex natural images. This embodiment combines the low-rank Hankel matrix optimization of ADMM and uses its redundancy characteristics to capture the global structure of the image (such as edges and textures) to avoid generating blur or distortion.

[0175] (4) Existing methods usually process images at a fixed scale, resulting in insufficient detail recovery. This embodiment constructs multi-scale samples, trains and generates models at different scales to avoid local optima, and adopts an alternating optimization strategy (ADMM+DC) to achieve cross-scale information complementarity.

[0176] (5) For various degradation types such as noise, pixel loss, and mosaic, this embodiment combines prior generation with low-rank constraints to adapt to different degradation modes without adjusting the model structure, thereby improving generalization ability.

[0177] like Figure 4 Some embodiments of this application provide an image restoration apparatus, the apparatus comprising:

[0178] The image acquisition module 1001 is used to acquire the degraded image to be trained;

[0179] The first tensor generation module 1002 is used to extract multiple first image patches from the degraded image to be trained, and extract the first Hankel matrix corresponding to each of the multiple first image patches, and fold each first Hankel matrix into multiple corresponding first tensors according to different scales.

[0180] The model training module 1003 is used to input multiple first tensors corresponding to each first Hankel matrix into a preset generation-diffusion model, so that the generation-diffusion model learns the prior distribution of the image using stochastic differential equations, and obtains the trained generation-diffusion model based on the prior distribution of the image.

[0181] The degraded image acquisition module 1004 is used to acquire the target degraded image in response to the recovery signal of the target degraded image;

[0182] The second tensor generation module 1005 is used to extract multiple second image patches of the target degraded image, and extract the second Hankel matrix corresponding to each second image patch, and fold each second Hankel matrix into a variety of corresponding second tensors according to different scales.

[0183] The target image restoration module 1006 is used to restore the target image from the multiple second tensors corresponding to each second Hankel matrix according to the generation diffusion model.

[0184] It should be noted that the image restoration device provided in this embodiment is based on the same inventive concept as the image restoration method described above. Therefore, the content of the image restoration method described above is also applicable to the content of this device, and will not be repeated here.

[0185] like Figure 5 This application also provides an electronic device, which includes:

[0186] At least one hydrogen fuel cell;

[0187] At least one memory;

[0188] At least one processor;

[0189] At least one program;

[0190] The program is stored in memory, and the processor executes at least one program to implement the image restoration method described above in this disclosure.

[0191] This electronic device can be any smart terminal, including mobile phones, tablets, personal digital assistants (PDAs), and in-vehicle computers.

[0192] The electronic devices according to embodiments of this application will now be described in detail.

[0193] The processor 1600 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this disclosure.

[0194] The memory 1700 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1700 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1700 and is called and executed by the processor 1600 to perform an image restoration method according to an embodiment of this disclosure.

[0195] The input / output interface 1800 is used to implement information input and output.

[0196] The communication interface 1900 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0197] Bus 2000 transmits information between various components of the device (e.g., processor 1600, memory 1700, input / output interface 1800, and communication interface 1900);

[0198] The processor 1600, memory 1700, input / output interface 1800 and communication interface 1900 are connected to each other within the device via bus 2000.

[0199] This disclosure also provides a storage medium, which is a computer-readable storage medium storing computer-executable instructions for causing a computer to perform the image restoration method described above.

[0200] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0201] The embodiments described in this disclosure are for the purpose of more clearly illustrating the technical solutions of this disclosure and do not constitute a limitation on the technical solutions provided by this disclosure. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by this disclosure are also applicable to similar technical problems.

[0202] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this disclosure, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0203] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0204] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0205] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0206] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0207] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0208] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0209] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0210] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0211] The above is a detailed description of the preferred embodiments of this application. However, the embodiments of this application are not limited to the above-described implementation methods. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the embodiments of this application. All such equivalent modifications or substitutions are included within the scope defined by the claims of the embodiments of this application.

[0212] The embodiments of this application have been described in detail above with reference to the accompanying drawings. However, this application is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of this application.

Claims

1. An image restoration method, characterized in that, The method includes: Obtain the degraded image to be trained; Extract multiple first image patches from the degraded image to be trained, and extract the first Hankel matrix corresponding to each of the multiple first image patches. Fold each first Hankel matrix into multiple corresponding first tensors according to different scales. Each of the first Hankel matrix and the corresponding multiple first tensors are input into a preset generative diffusion model, so that the generative diffusion model learns the prior distribution of the image using stochastic differential equations, and obtains the trained generative diffusion model based on the prior distribution of the image. In response to the recovery signal of the target degraded image, the target degraded image is acquired; Extract multiple second image patches from the target degraded image, and extract the second Hankel matrix corresponding to each second image patch. Fold each second Hankel matrix into multiple corresponding second tensors according to different scales. The target image is recovered from the various second tensors corresponding to each second Hankel matrix according to the generative diffusion model.

2. The image restoration method according to claim 1, characterized in that, The step of recovering the target image from the multiple second tensors corresponding to each second Hankel matrix according to the generative diffusion model includes: Perform M iterations on the various second tensors corresponding to the second Hankel matrix to obtain the third image patch after the Mth iteration; where M is an integer greater than or equal to 2; The target image is generated based on the third image block; The process of the i-th iteration includes: Determine the multiple tensors of the current input; when i is 1, the multiple tensors of the current input are the multiple second tensors; and when i is any number from 2 to M, the multiple tensors of the current input are multiple tensors obtained by first performing Hankel matrix transformation on the third image patch after the (i-1)th iteration, and then folding the transformed Hankel matrix according to the different scales. Using the inverse equation corresponding to the stochastic differential equation, multiple candidate recovery samples that conform to the prior distribution of the image are sampled from the various tensors of the current input; The multiple candidate recovery samples are expanded into multiple corresponding Hankel matrices; Based on the expanded Hankel matrices, the third image patch is generated after the i-th iteration.

3. The image restoration method according to claim 2, characterized in that, The step of generating the third image patch after the i-th iteration based on the expanded multiple Hankel matrices includes: The weighted mean of multiple expanded Hankel matrices is used to obtain the weighted mean Hankel matrix. Based on the weighted average Hankel matrix, the third image patch is generated after the i-th iteration.

4. The image restoration method according to claim 3, characterized in that, After sampling multiple candidate recovery samples that conform to the prior image distribution from the current multiple tensors using the inverse equation corresponding to the stochastic differential equation, the method further includes: The multiple candidate recovery samples are corrected to obtain corrected multiple candidate recovery samples; Expanding the multiple candidate recovery samples into corresponding multiple Hankel matrices includes: The corrected candidate recovery samples are expanded into corresponding Hankel matrices.

5. The image restoration method according to claim 4, characterized in that, The step of generating the third image patch after the i-th iteration based on the weighted average Hankel matrix includes: The weighted mean Hankel matrix is ​​optimized by performing low-rank optimization using the alternating direction multiplier algorithm to obtain the low-rank optimized Hankel matrix. Based on the low-rank optimized Hankel matrix, the third image patch is generated after the i-th iteration.

6. The image restoration method according to claim 5, characterized in that, After generating the third image patch after the i-th iteration based on the Hankel matrix optimized by the low rank, the method further includes: DC consistency correction is performed on the third image block after the i-th iteration.

7. The image restoration method according to claim 1, characterized in that, The generative diffusion model is a denoising score matching loss function.

8. An image restoration device, characterized in that, The device includes: The image acquisition module is used to acquire the degraded image to be trained; The first tensor generation module is used to extract multiple first image patches from the degraded image to be trained, and extract the first Hankel matrix corresponding to each of the multiple first image patches, and fold each first Hankel matrix into multiple corresponding first tensors according to different scales. The model training module is used to input the various first tensors corresponding to each first Hankel matrix into a preset generation-diffusion model, so that the generation-diffusion model learns the prior distribution of the image using stochastic differential equations, and obtains the trained generation-diffusion model based on the prior distribution of the image. A degraded image acquisition module is used to acquire the target degraded image in response to a recovery signal of the target degraded image; The second tensor generation module is used to extract multiple second image blocks of the target degraded image, extract the second Hankel matrix corresponding to each second image block, and fold each second Hankel matrix into multiple corresponding second tensors according to different scales. The target image recovery module is used to recover the target image from the multiple second tensors corresponding to each second Hankel matrix according to the generation diffusion model.

9. An electronic device, characterized in that: The device includes at least one control processor and a memory for communicatively connecting to the at least one control processor; the memory stores instructions executable by the at least one control processor to enable the at least one control processor to perform an image restoration method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions for causing a computer to perform an image restoration method according to any one of claims 1 to 7.

Images