Image super-resolution method and device based on conditional generation diffusion model

Abstract

The invention provides an image super-resolution method and device based on a conditional generation diffusion model, and the method comprises the steps: carrying out the bicubic linear difference of a to-be-processed low-resolution image, and obtaining an interpolation image; inputting the interpolation image into a generator to obtain a guide image; performing subtraction on the interpolation image and the guide image to obtain disturbance distribution; reforming the disturbance distribution to obtain first Gaussian distribution; matching the standard deviation of the first Gaussian distribution with a preset noise adding sequence, and determining a registration noise adding step number; splicing the interpolation image and the guide image in the channel dimension to obtain a noisy image; and inputting the noisy image and the registration noise adding step number into the trained noise prediction neural network to obtain the noise of the noisy image. The invention provides an image super-resolution method and device based on a conditional generation diffusion model. The method and device are used for solving the problems that in the prior art, image details cannot be accurately restored, optimization is unstable, a mode is prone to collapse and collapse, and training is complex.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to an image super-resolution method and device based on a conditional generative diffusion model. BACKGROUND

[0002] Low-field nuclear magnetic resonance (LF-NMR) imaging is an advanced modality that can provide valuable information about water status and distribution and internal structure, helping non-destructive detection of fruit quality and damage level. However, the extraction of high-resolution images from LF-NMR scans requires a long acquisition time, hindering the efficiency of the evaluation. The reason for the slow acquisition rate of low-frequency nuclear magnetic resonance is that the reduction of acquisition time is proportional to the reduction of spatial resolution. For example, if the LF-NMR images are acquired in half the resolution in the in-plane, the acquisition time will actually be reduced by half. Nevertheless, it is still challenging to achieve high-precision non-destructive detection when using low-resolution low-frequency nuclear magnetic resonance imaging images. Therefore, it is of great value to develop a method for improving the spatial resolution of LF-NMR images while maintaining a relatively short acquisition time, which is of great value for reducing the inconvenience of non-destructive detection.

[0003] Super-resolution is responsible for improving the spatial resolution of low-resolution images, which can restore image quality after fast low-frequency nuclear magnetic resonance scanning. Traditional super-resolution methods rely on linear interpolation and optimization-based methods to achieve resolution enhancement. However, with the development of deep learning, deep learning-driven super-resolution methods have surpassed traditional methods and shown superior performance, but there are still some uncertainties in these methods. Initially, super-resolution convolutional neural networks were designed to generate high-resolution images. Subsequently, in order to further enhance the performance of the reconstructed images, dense blocks and residual blocks were introduced to enrich the super-resolution convolutional neural network architecture. Despite these advances, they have not been fully applied to MR datasets due to their poor de-aliasing ability and expensive computational cost. It was not until the generation of adversarial networks that convolutional neural networks were completely changed, and they showed superior performance in many image high-resolution tasks. However, GAN-based super-resolution models have a higher risk of collapse. The generator may not be able to extract the diversity and complexity of the underlying data distribution, but instead repeatedly generate a series of prototype examples; at the same time, the objective function of the GAN-based high-resolution model is non-convex, making it difficult to find a stable convergence point, thereby disturbing the entire training process, and these limitations hinder the use of GAN for LF-NMR images.

[0004] In summary, the existing LF-NMR image super-resolution method has the problems of being unable to accurately restore image details, unstable optimization, prone to collapse, and complex training. SUMMARY

[0005] This invention provides an image super-resolution method and apparatus based on a conditional generation diffusion model, which solves the problems existing in the prior art, such as inability to accurately restore image details, unstable optimization, easy collapse of the model, and complex training.

[0006] On one hand, the present invention provides an image super-resolution method based on a conditional generation diffusion model, comprising:

[0007] S1: Perform bicubic linear interpolation on the low-resolution image to be processed to obtain an interpolated image, wherein the resolution of the interpolated image is the target resolution, and the target resolution is the resolution of the target image;

[0008] S2: Input the interpolated image into the generator to obtain a guide image, the resolution of which is the target resolution;

[0009] S3: Subtract the interpolated image from the guided image to obtain the perturbation distribution;

[0010] S4: Reshape the disturbance distribution to obtain a first Gaussian distribution with a mean of 0;

[0011] S5: Match the standard deviation of the first Gaussian distribution with the preset noise-adding sequence to determine the number of registration and noise-adding steps;

[0012] S6: The interpolated image and the guiding image are stitched together along the channel dimension to obtain a noisy image;

[0013] S7: Input the noisy image and the number of registration and noise addition steps into the trained noise prediction neural network to obtain the noise of the noisy image. The noise prediction neural network is trained based on the forward noise addition process.

[0014] S8: Calculate the initial image based on the noisy image and its noise;

[0015] S9: Calculate the mean and variance of the noisy image from the previous step based on the noisy image and the initial image;

[0016] S10: Randomly sample from the standard normal distribution to generate a second Gaussian distribution. Obtain the previous noisy image based on the mean and variance of the previous noisy image and the second Gaussian distribution. The second Gaussian distribution has a mean of 0, a variance of 1, and its resolution is the same as the target resolution.

[0017] S11: Take the noisy image from the previous step as the new noisy image, subtract one from the registration and noise addition step number and take it as the new registration and noise addition step number. Determine whether the new registration and noise addition step number is 0. If it is, the new noisy image is the target image. If not, continue to execute S7-S11.

[0018] Optionally, the reorganizing the disturbance distribution to obtain a first Gaussian distribution comprises:

[0019] The disturbance distribution is reorganized according to the following formula:

[0020]

[0021] wherein ζ is the disturbance distribution, is the first Gaussian distribution.

[0022] Optionally, the matching the standard deviation of the first Gaussian distribution with a preset noise sequence to determine a registration noise step number comprises:

[0023] The registration noise step number is determined according to the following formula:

[0024]

[0025] wherein t m is the registration noise step number, β i = 1-α i , α i belongs to the preset noise sequence {α1, α2, …, α T-1 ,α T}, represents the standard deviation of Gaussian noise when the i-th step of noise is added, T is the maximum noise step number, σ is the standard deviation of the first Gaussian distribution, and p is the order of the norm.

[0026] Optionally, the training of the noise prediction neural network based on a forward noise adding process comprises:

[0027] S71: selecting an initial high-resolution image X0 from a training set;

[0028] S72: randomly selecting an integer t in [1, T] as a current noise step number, and obtaining a current noise sequence {α1, α2, …, α T-1 ,α T} from the preset noise sequence {α1, α2, …, α t-1 ,α t} according to the current noise step number t;

[0029] S73: gradually adding noise to the initial high-resolution image X0 according to an initial Gaussian noise ∈ and the current noise sequence {α1, α2, …, α t-1 ,α t} to obtain a noisy high-resolution image X t , wherein the initial Gaussian noise ∈ is obtained by randomly sampling from a standard normal distribution;

[0030] S74: obtaining the current noise-added step t and the noisy high-resolution image X t inputting the noise prediction neural network to obtain the noisy high-resolution image X t θ ;

[0031] S75: optimizing the parameters of the noise prediction neural network based on a loss function and a back propagation algorithm, the loss function being constructed based on the initial Gaussian noise ∈ and the noise ∈ of the noisy high-resolution image X t θ .

[0032] S76: determining whether a training termination condition is met, if yes, stopping the training, and if not, executing S71-S76.

[0033] Optionally, the step of gradually adding noise to the initial high-resolution image X0 based on the initial Gaussian noise ∈ and the current noise-added sequence {α1, α2, …, α t-1 ,α t} to obtain the noisy high-resolution image X t comprises:

[0034] adding noise to the initial high-resolution image X0 according to the following formula:

[0035]

[0036] wherein,

[0037] Optionally, the loss function is:

[0038]

[0039] Optionally, the step of calculating the initial image based on the noisy image and the noise thereof is implemented according to the following formula:

[0040]

[0041] wherein, X0 ′ is the initial image, X ′ m is the noisy image, m is the registration noise-added step, ∈ ′ θ is the noise of the noisy image.

[0042] Optionally, the step of calculating the mean and variance of the previous-step noisy image based on the noisy image and the initial image is implemented according to the following two formulas:

[0043] ​​

[0044] wherein μ m-1 is the mean of the noisy image of the previous step, is the variance of the noisy image of the previous step.

[0045] Optionally, the noisy image of the previous step is:

[0046]

[0047] wherein X′ m-1 is the noisy image of the previous step, and Z is the second Gaussian distribution.

[0048] In another aspect, the present application provides an image super-resolution device based on a conditional generative diffusion model, comprising an interpolation module, a guide module, a perturbation module, a regularization module, a registration module, a splicing module, a prediction module, a first calculation module, a second calculation module, a third calculation module, and an iterative output module, wherein:

[0049] The interpolation module is used for bicubic linear difference value to the low-resolution image to be processed to obtain an interpolated image, the resolution of the interpolated image is a target resolution, and the target resolution is the resolution of a target image.

[0050] The guide module is used for inputting the interpolated image into a generator to obtain a guide image, and the resolution of the guide image is the target resolution.

[0051] The perturbation module is used for subtracting the interpolated image and the guide image to obtain a perturbation distribution.

[0052] The regularization module is used for regularizing the perturbation distribution to obtain a first Gaussian distribution, and the mean of the first Gaussian distribution is 0.

[0053] The registration module is used for matching the standard deviation of the first Gaussian distribution with a preset noise sequence to determine a registration noise step number.

[0054] The splicing module is used for splicing the interpolated image and the guide image in the channel dimension to obtain a noisy image.

[0055] The prediction module is used for inputting the noisy image and the registration noise step number into a trained noise prediction neural network to obtain the noise of the noisy image, and the noise prediction neural network is trained based on a forward noise process.

[0056] The first calculation module is used for calculating an initial image based on the noisy image and the noise thereof.

[0057] The second calculation module is used for calculating the mean and the variance of the noisy image of the previous step based on the noisy image and the initial image.

[0058] The third calculation module is configured to generate a second Gaussian distribution by random sampling from a standard normal distribution, and calculate the last-step noisy image based on a mean value and a variance of the last-step noisy image and the second Gaussian distribution, wherein the second Gaussian distribution has a mean value of 0 and a variance of 1 and has a same resolution as the target resolution.

[0059] The iteration output module is configured to take the last-step noisy image as a new noisy image, take a new registration noise step number obtained by subtracting one from the registration noise step number as the new registration noise step number, judge whether the new registration noise step number is 0, if yes, the new noisy image is the target image, and if not, jump to the prediction module.

[0060] The above scheme reduces the number of iterations required by the diffusion model by introducing a guide image as a condition to determine the registration noise step number, thereby improving the generation rate and quality of the entire model. Compared with other deep learning methods, the above scheme avoids the problems of excessive model parameters, model collapse and model collapse. There is a structural difference between the existing interpolation-based method, the reconstruction-based method and the deep learning-based method, which solves the problems of inaccurate image detail restoration, unstable optimization, model collapse and training complexity of the existing methods. Therefore, the model can be trained more stably and quickly, and a clear high-resolution image can be accurately and quickly obtained, thereby improving the quality of the low-resolution image and promoting the application of low-field nuclear magnetic instruments in the quality detection field. BRIEF DESCRIPTION OF DRAWINGS

[0061] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0062] Figure 1 A flowchart of an image super-resolution method based on a conditional generation diffusion model provided by an embodiment of the present application is shown.

[0063] Figure 2 A structural diagram of a noise prediction neural network provided by an embodiment of the present application is shown.

[0064] Figure 3 A principle diagram of a forward noise adding process and a reverse noise removing process provided by an embodiment of the present application is shown.

[0065] Figure 4 A structural diagram of an image super-resolution device based on a conditional generation diffusion model provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0066] In order to make the objects, technical solutions and advantages of the present application clearer, the following further describes the present application in detail with reference to the drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, and not all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0067] It should be noted that the terms "first", "second", and the like involved in the documents of the present application are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0068] Figure 1 The flow of the image super-resolution method based on the condition generation diffusion model provided by the embodiments of the present application is shown in detail, which includes:

[0069] S1: bicubic linear difference is performed on the low-resolution image to be processed to obtain an interpolation image, and the resolution of the interpolation image is a target resolution, and the target resolution is the resolution of a target image;

[0070] In an example, the low-resolution image to be processed is a blurred low-field nuclear magnetic resonance (LF-NMR) image.

[0071] In an example, the target resolution is 256*256.

[0072] S2: the interpolation image is input into a generator to obtain a guide image, and the resolution of the guide image is the target resolution;

[0073] The generator is composed of an encoder for down-sampling and a decoder for up-sampling, and realizes a simple super-resolution process.

[0074] It should be noted that the interpolation image and the guide image, although having the same resolution as the target image, have lower clarity than the target image, and the clarity of the interpolation image is lower than that of the guide image.

[0075] The guide image will guide the inverse denoising process as a condition of the condition generation diffusion model.

[0076] S3: difference between the interpolation image and the guide image to obtain a perturbation distribution;

[0077] S4: reorganize the perturbation distribution to obtain a first Gaussian distribution, the mean of the first Gaussian distribution being 0;

[0078] In an example, reorganizing the perturbation distribution to obtain the first Gaussian distribution comprises:

[0079] The perturbation distribution is reorganized according to the following formula:

[0080]

[0081] wherein ζ is the perturbation distribution, is the first Gaussian distribution.

[0082] S5: match the standard deviation of the first Gaussian distribution with a preset noise sequence to determine a registration noise step number;

[0083] In an example, matching the standard deviation of the first Gaussian distribution with the preset noise sequence to determine the registration noise step number comprises:

[0084] The registration noise step number is determined according to the following formula:

[0085]

[0086] wherein t m is the registration noise step number, β i = 1-α i , α i belongs to the preset noise sequence {α1, α2, …, α T-1 , α T}, represents the standard deviation of Gaussian noise when the i-th step of noise is added, T is the maximum noise step number, σ is the standard deviation of the first Gaussian distribution, and p is the order of the norm.

[0087] By determining the registration noise step number, the number of iterations required by the diffusion model is reduced, thereby improving the generation rate and quality of the entire model. Compared with other deep learning methods, the problems of excessive model parameters, easy model collapse and easy model collapse are avoided.

[0088] S6: concatenate the interpolation image and the guide image in the channel dimension to obtain a noisy image;

[0089] Specifically, the concatenation in the channel dimension is realized by a Concat operation.

[0090] S7: input the noisy image and the registration noise step number into a trained noise prediction neural network to obtain the noise of the noisy image, the noise prediction neural network being trained based on a forward noise adding process;

[0091] In one example, the noise prediction neural network is built upon the UNet neural network, and its structure is as follows: Figure 2 As shown, the encoder consists of a downsampling module and an upsampling module.

[0092] In one example, training the noise prediction neural network based on the forward noise addition process includes:

[0093] S71: Select an initial high-resolution image X0 from the training set;

[0094] S72: Randomly select an integer t from [1,T] as the current noise-adding step number, and based on the current noise-adding step number t, start from the preset noise-adding sequence {α1,α2,…,α...} T-1 ,α T Obtain the current noisy sequence {α1, α2, ..., α} from} t-1 ,α t};

[0095] S73: Based on the initial Gaussian noise ∈ and the current noisy sequence {α1, α2, ..., α...} t-1 ,α t The initial high-resolution image X0 is gradually noise-added to obtain a noisy high-resolution image X. t The initial Gaussian noise ∈ is obtained by random sampling from a standard normal distribution;

[0096] It should be noted that S73 is the forward noise addition process.

[0097] S74: Combine the current noise-adding step number t and the noisy high-resolution image X t Input a noisy prediction neural network to obtain a noisy high-resolution image X. t noise ∈ θ ;

[0098] S75: Optimizing the parameters of a noisy prediction neural network based on a loss function and backpropagation algorithm. The loss function is based on the initial Gaussian noise ∈ and the noisy high-resolution image X. t noise ∈ θ Build;

[0099] S76: Determine if the training termination condition is met. If it is met, stop training. If it is not met, execute S71 to S76.

[0100] In one example, the initial high-resolution image X0 is a high-resolution NMR image.

[0101] Furthermore, based on the initial Gaussian noise ∈ and the current noisy sequence {α1, α2, ..., α...} t-1 ,α t The initial high-resolution image X0 is gradually noise-added to obtain a noisy high-resolution image x.t comprises:

[0102] The initial high-resolution image X0 is added with noise according to the following formula:

[0103]

[0104] wherein,

[0105] It should be noted that in fact, when adding noise at each step, the noise adding formula is as follows:

[0106]

[0107] wherein, ∈ i is the Gaussian noise when adding noise at the i-th step;

[0108] Considering that the entire noise adding process is a continuous Markov process, and the noise added is sampled from a Gaussian distribution, and in view of the superposition of Gaussian noise, the entire noise adding process can be simplified to one-time noise adding by parameter renormalization, and the above formula for adding noise to the initial high-resolution image X0 is obtained.

[0109] Further, the loss function is:

[0110]

[0111] In an example, the training termination condition is that a preset number of training times is reached.

[0112] S8: Calculate the initial image based on the noisy image and the noise thereof;

[0113] In an example, calculating the initial image based on the noisy image and the noise thereof is implemented according to the following formula:

[0114]

[0115] wherein, X0 ′ is the initial image, X ′ m is the noisy image, and m is the number of registration noise adding steps, ∈ ′ θ is the noise of the noisy image.

[0116] S9: Calculate the mean and variance of the noisy image of the previous step based on the noisy image and the initial image;

[0117] In an example, calculating the mean and variance of the noisy image of the previous step based on the noisy image and the initial image is implemented according to the following two formulas:

[0118]

[0119] wherein μ m-1 is the mean of the previous noisy image, is the variance of the previous noisy image.

[0120] S10: randomly sampling from a standard normal distribution to generate a second Gaussian distribution, calculating the previous noisy image based on the mean and variance of the previous noisy image and the second Gaussian distribution, the mean of the second Gaussian distribution being 0 and the variance being 1, and the resolution of the second Gaussian distribution being the same as the target resolution;

[0121] In an example, the previous noisy image is:

[0122]

[0123] wherein X' m-1 is the previous noisy image, and Z is the second Gaussian distribution.

[0124] It should be noted that calculating the previous noisy image based on the mean and variance of the previous noisy image and the second Gaussian distribution is actually:

[0125] The mean and variance of the second Gaussian distribution are changed based on the mean and variance of the previous noisy image, and the previous noisy image is obtained.

[0126] S11: taking the previous noisy image as a new noisy image, taking the registration noise adding step number minus one as a new registration noise adding step number, judging whether the new registration noise adding step number is 0, if yes, the new noisy image is the target image, if not, continuing to execute S7-S11.

[0127] It should be noted that S7-S11 realizes a reverse denoising process.

[0128] Figure 3 The principle of the forward noise adding process and the reverse denoising process is shown, and in combination with a Markov chain process, the forward noise adding process is expressed as a prior distribution, and the reverse denoising process is expressed as a posterior distribution.

[0129] The scheme reduces the number of iterations required by the diffusion model by introducing a guide image as a condition to determine the registration noise step number, thereby improving the generation rate and quality of the entire model. Compared with other deep learning methods, the problems of excessive model parameters, model collapse and model collapse are avoided. There is a structural difference between the existing interpolation-based method, the reconstruction-based method and the deep learning-based method, which solves the problems of inaccurate image detail restoration, unstable optimization, model collapse and complex training, so that the model can be trained more stably and quickly, and the clear high-resolution image can be accurately and quickly obtained, the quality of the low-resolution image is improved, and the application of low-field nuclear magnetic instrument in the quality detection field is promoted.

[0130] Based on the same technical concept, Figure 4 The structure of the image super-resolution device based on the conditional generation diffusion model provided by the embodiment of the application is shown, which comprises an interpolation module, a guide module, a disturbance module, a regularization module, a registration module, a splicing module, a prediction module, a first calculation module, a second calculation module, a third calculation module and an iteration output module, wherein:

[0131] The interpolation module is used for bicubic linear difference of the low-resolution image to be processed to obtain an interpolation image, and the resolution of the interpolation image is a target resolution, and the target resolution is the resolution of the target image;

[0132] The guide module is used for inputting the interpolation image into a generator to obtain a guide image, and the resolution of the guide image is the target resolution;

[0133] The disturbance module is used for subtracting the interpolation image and the guide image to obtain a disturbance distribution;

[0134] The regularization module is used for regularizing the disturbance distribution to obtain a first Gaussian distribution, and the mean of the first Gaussian distribution is 0;

[0135] The registration module is used for matching the standard deviation of the first Gaussian distribution with a preset noise sequence to determine a registration noise step number;

[0136] The splicing module is used for splicing the interpolation image and the guide image in the channel dimension to obtain a noisy image;

[0137] The prediction module is used for inputting the noisy image and the registration noise step number into a trained noise prediction neural network to obtain the noise of the noisy image, and the noise prediction neural network is trained based on a forward noise process;

[0138] The first calculation module is used for calculating an initial image based on the noisy image and the noise thereof;

[0139] The second calculation module is used for calculating the mean and variance of the noisy image of the previous step based on the noisy image and the initial image.

[0140] The third calculation module is configured to generate a second Gaussian distribution by random sampling from a standard normal distribution, calculate the noisy image of the previous step based on the mean and variance of the noisy image of the previous step and the second Gaussian distribution, and the mean of the second Gaussian distribution is 0 and the variance is 1, and the resolution of the second Gaussian distribution is the same as the target resolution.

[0141] The iteration output module is configured to take the noisy image of the previous step as a new noisy image, take the registration noise adding step number minus one as a new registration noise adding step number, judge whether the new registration noise adding step number is 0, if yes, the new noisy image is the target image, and if not, jump to the prediction module.

[0142] Based on the same technical concept, the embodiment of the present application provides a computing device, comprising:

[0143] a memory configured to store program instructions;

[0144] a processor configured to invoke the program instructions stored in the memory, and execute the above method according to the obtained program.

[0145] Based on the same technical concept, the embodiment of the present application provides a computer readable nonvolatile storage medium, comprising computer readable instructions, when the computer reads and executes the computer readable instructions, the computer executes the above method.

[0146] The present application is described with reference to flowcharts and / or block diagrams of the method, device (system) and computer program product according to the embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of the flows and / or blocks in the flowchart and / or block diagram can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device that realizes the functions specified in the flowchart and / or block diagram. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that realizes the functions specified in one or more flows and / or blocks.

[0147] These computer program instructions can also be stored in a computer readable memory capable of guiding the computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer readable memory produce a product including instruction apparatus, which realizes the functions specified in the flowchart and / or block diagram. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that realizes the functions specified in one or more flows and / or blocks.

[0148] These computer program instructions can also be loaded into a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 one or more flowcharts and / or blocks

[0149] Although preferred embodiments of the application have been described herein, it will be apparent to those skilled in the art that various modifications can be made within the scope of the application without departing from the spirit of the application. Accordingly, it is intended that all such possible modifications be included within the scope of the application as defined in the following claims in which the use of the singular is deemed to include the plural, unless specifically stated otherwise.

[0150] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described herein.

Claims

1. An image super-resolution method based on a conditional generative diffusion model, characterized in that, include: S1: Perform bicubic linear interpolation on the low-resolution image to be processed to obtain an interpolated image, wherein the resolution of the interpolated image is the target resolution, and the target resolution is the resolution of the target image; S2: Input the interpolated image into the generator to obtain a guide image, the resolution of which is the target resolution; S3: Subtract the interpolated image from the guided image to obtain the perturbation distribution; S4: Reshape the disturbance distribution to obtain a first Gaussian distribution with a mean of 0. S5: Match the standard deviation of the first Gaussian distribution with the preset noise-adding sequence to determine the number of registration and noise-adding steps; S6: The interpolated image and the guiding image are stitched together in the channel dimension to obtain a noisy image; S7: Input the noisy image and the number of registration and noise addition steps into the trained noise prediction neural network to obtain the noise of the noisy image. The noise prediction neural network is trained based on the forward noise addition process. S8: Calculate the initial image based on the noisy image and its noise; S9: Calculate the mean and variance of the noisy image from the previous step based on the noisy image and the initial image; S10: Randomly sample from the standard normal distribution to generate a second Gaussian distribution. Obtain the noisy image from the previous step based on the mean and variance of the noisy image from the previous step and the second Gaussian distribution. The second Gaussian distribution has a mean of 0, a variance of 1, and its resolution is the same as the target resolution. S11: Take the noisy image from the previous step as the new noisy image, subtract one from the registration and noise addition step number and take it as the new registration and noise addition step number. Determine whether the new registration and noise addition step number is 0. If it is, the new noisy image is the target image. If not, continue to execute S7-S11.

2. The method according to claim 1, characterized in that, The process of reshaping the perturbation distribution to obtain the first Gaussian distribution includes: The perturbation distribution is reshaped according to the following formula: Where ζ represents the disturbance distribution. It is the first Gaussian distribution.

3. The method according to claim 1, characterized in that, The step of matching the standard deviation of the first Gaussian distribution with the preset noise-adding sequence to determine the number of registration and noise-adding steps includes: The number of registration and noise addition steps is determined according to the following formula: Among them, t m The number of registration noise addition steps, β i =1-α i α i Belonging to the preset noise-adding sequence {α1, α2, ..., α T-1 ,α T } represents the standard deviation of the Gaussian noise when adding noise in the i-th step, T is the maximum number of noise-adding steps, σ is the standard deviation of the first Gaussian distribution, and p is the order of the norm.

4. The method according to claim 1, characterized in that, The noise prediction neural network is trained based on the forward noise addition process, including: S71: Select an initial high-resolution image X0 from the training set; S72: Randomly select an integer t from [1,T] as the current noise-adding step number, and based on the current noise-adding step number t, start from the preset noise-adding sequence {α1,α2,…,α...} T-1 ,α T Obtain the current noisy sequence {α1, α2, ..., α} from} t-1 ,α t }; S73: Based on the initial Gaussian noise ∈ and the current noise-added sequence {α1, α2, ..., α...} t-1 ,α t The initial high-resolution image X0 is gradually noise-added to obtain a noisy high-resolution image X. t The initial Gaussian noise ∈ is obtained by random sampling from a standard normal distribution; S74: Combine the current noise-adding step number t with the noisy high-resolution image X t The noisy high-resolution image X is obtained by inputting the noise prediction neural network. t noise ∈ θ ; S75: Optimize the parameters of the noise prediction neural network based on a loss function and a backpropagation algorithm, wherein the loss function is based on the initial Gaussian noise ∈ and the noisy high-resolution image X. t noise ∈ θ Build; S76: Determine if the training termination condition is met. If it is met, stop training. If it is not met, execute S71 to S76.

5. The method according to claim 4, characterized in that, The initial Gaussian noise ∈ and the current noise-added sequence {α1, α2, ..., α} are used. t-1 ,α t The initial high-resolution image X0 is gradually noise-added to obtain a noisy high-resolution image X. t include: Noise is added to the initial high-resolution image X0 according to the following formula: in, 6. The method according to claim 4, characterized in that, The loss function is:

7. The method according to claim 1, characterized in that, The calculation of the initial image based on the noisy image and its noise is achieved according to the following formula: Where X′0 is the initial image, X′ m The image with noise is denoised as m, and the number of registration and noise addition steps is m. ∈′ θ The noise in the noisy image.

8. The method according to claim 7, characterized in that, The calculation of the mean and variance of the noisy image from the previous step based on the noisy image and the initial image is achieved using the following two formulas: Where, μ m-1 The mean of the noisy image from the previous step. Let V be the variance of the noisy image from the previous step.

9. The method according to claim 8, characterized in that, The noisy image from the previous step is: Among them, X ′ m-1 Let Z be the noisy image from the previous step, and Z be the second Gaussian distribution.

10. An image super-resolution device based on a conditional generation diffusion model, characterized in that, It includes an interpolation module, a guidance module, a perturbation module, a renormalization module, a registration module, a stitching module, a prediction module, a first calculation module, a second calculation module, a third calculation module, and an iterative output module, wherein: The interpolation module is used to perform bicubic linear interpolation on the low-resolution image to be processed to obtain an interpolated image. The resolution of the interpolated image is the target resolution, and the target resolution is the resolution of the target image. The guiding module is used to input the interpolated image into the generator to obtain a guiding image, the resolution of which is the target resolution; The perturbation module is used to subtract the interpolated image and the guiding image to obtain the perturbation distribution; The reshaping module is used to reshape the disturbance distribution to obtain a first Gaussian distribution with a mean of 0. The registration module is used to match the standard deviation of the first Gaussian distribution with a preset noise-adding sequence to determine the number of registration and noise-adding steps; The stitching module is used to stitch the interpolated image and the guiding image in the channel dimension to obtain a noisy image; The prediction module is used to input the noisy image and the number of registration and noise addition steps into a trained noise prediction neural network to obtain the noise of the noisy image. The noise prediction neural network is trained based on the forward noise addition process. The first calculation module is used to calculate an initial image based on the noisy image and its noise; The second calculation module is used to calculate the mean and variance of the noisy image from the previous step based on the noisy image and the initial image; The third calculation module is used to randomly sample from the standard normal distribution to generate a second Gaussian distribution, and calculate the previous noisy image based on the mean and variance of the previous noisy image and the second Gaussian distribution. The second Gaussian distribution has a mean of 0, a variance of 1, and its resolution is the same as the target resolution. The iterative output module is used to take the previous noisy image as the new noisy image, subtract one from the registration and noise addition step number and take it as the new registration and noise addition step number, and determine whether the new registration and noise addition step number is 0. If it is, the new noisy image is the target image; otherwise, it jumps to the prediction module.

Images