Data consistency training method of diffusion model for image inpainting, image inpainting method and device

Abstract

The application provides a diffusion model data consistency training method, an image restoration method and equipment for image restoration. The method comprises the following steps: in the data consistency training process of the diffusion model for image restoration, obtaining training input data of the diffusion model at the tth step; the data is intermediate image data obtained after an input image to be restored is taken as the starting point of the reverse denoising process and iterated to the tth step; inputting the training input data at the tth step and information into the core network of the diffusion model to obtain a prediction result, and combining the supervised data to calculate a loss function to update the parameters of the core network; taking the intermediate image data as the training input data, so that the training process optimizes the single-step denoising error and optimizes the cumulative error generated when the multi-step denoising iteration is performed in the test stage. The application can effectively solve the problems of high cumulative error, large calculation overhead and low adaptability of the existing image restoration training method.

Description

Technical Field

[0001] This application relates to the field of image restoration, and in particular to a data consistency training method for a diffusion model used for image restoration, as well as an image restoration method and device. Background Technology

[0002] Image inpainting technology is widely used in mobile device imaging, medical image processing, video surveillance optimization, and other fields. The core requirement is to restore high-quality, realistic images from low-quality images (including those with noise, blur, and insufficient resolution). Existing image inpainting techniques are mainly divided into two categories: non-deep learning schemes (such as those based on sparse coding, non-local means, and dark channel prior) and deep learning schemes. However, non-deep learning schemes are suitable for simple scenarios, but their inpainting accuracy and generalization ability are limited. Meanwhile, the denoising diffusion model (DDM), as a deep learning scheme, has shown excellent performance in tasks such as text-to-image synthesis and image style transfer. In recent years, it has been gradually extended to the field of image inpainting, including single-image super-resolution (SISR), image denoising, image deraining, and dual-camera super-resolution.

[0003] However, existing DDM models suffer from inconsistencies between training and testing processes, leading to uncontrolled cumulative errors. Traditional diffusion models, including DDM, only optimize single-round module errors during training, neglecting the cumulative errors introduced by the backward process during testing. The training input generated in the forward process differs fundamentally from the test input generated in the backward process, preventing the model from learning to correct cumulative errors during training. During testing, the errors gradually amplify, resulting in poor restoration performance. Secondly, there is the problem of high computational cost. If the data from the backward process is directly used for training to align with the testing flow, the input data must be calculated round by round from the final iteration step to the current iteration step, significantly increasing memory usage and time costs during training, making engineering deployment difficult. Finally, DDM-based image restoration models have limited generalization ability. Existing DDM-like models are mainly optimized for single image restoration tasks and rely on synthetic datasets for training, lacking adaptability to complex distortion problems in real-world scenarios. Therefore, there is an urgent need to design a training method specifically for image restoration. Summary of the Invention

[0004] In view of this, embodiments of this application provide a data consistency training method for a diffusion model for image inpainting, an image inpainting method, and an apparatus to eliminate or improve one or more defects existing in the prior art.

[0005] The first aspect of this application provides a data consistency training method for a diffusion model for image inpainting, the method comprising: During the data consistency training process for a diffusion model used for image inpainting, the training input data of the diffusion model at step t is obtained; wherein, the training input data is intermediate image data obtained after iterating to step t according to the inverse denoising process of the diffusion model, with an input image to be inpainted as the starting point; T represents the total number of denoising iterations of the diffusion model, and t represents an integer from T to 1. The training input data at step t and the information representing step t are input into the core network of the diffusion model to obtain the prediction result output by the core network. The loss function is calculated based on the prediction result and the supervision data corresponding to step t, and the parameters of the core network are updated according to the loss function; wherein, the intermediate image data obtained by the reverse denoising process is used as the training input data, so that the training process optimizes the single-step denoising error and the cumulative error generated by the diffusion model during multi-step denoising iteration in the testing phase.

[0006] In some embodiments of this application, obtaining the training input data of the diffusion model at step t includes: The total number of denoising iterations T is associated with the order of training rounds, and the training input data obtained in the k-th training round corresponds to the t-th step; where t = T - k + 1, and k represents an integer from 1 to T; In the first training round, i.e., at t=T, the input image to be repaired is used as the training input data; In the k-th training round, and when k≥2, the intermediate image data generated and cached in the (k-1)-th training round is loaded and reused as the training input data for the current t-th step.

[0007] In some embodiments of this application, it also includes: After completing the parameter update for the kth training round, the updated core network is used to process the training input data for the current t-th step, generate intermediate image data for the (t-1)th step, and cache it for use in the (k+1)th training round.

[0008] In some embodiments of this application, the supervision data corresponding to the t-th step represents the ground truth image data corresponding to the input image to be repaired, which is used as the optimization target for the prediction result.

[0009] In some embodiments of this application, the loss function includes at least one of mean squared error loss, KL divergence loss, and perceptual loss.

[0010] In some embodiments of this application, the data consistency training method for the diffusion model for image inpainting according to claim 1 is characterized in that the core network includes: a denoising network based on U-Net, a latent diffusion model, a residual offset network, or a Transformer architecture.

[0011] In some embodiments of this application, the image restoration includes at least one of: single-image super-resolution, image denoising, image deraining, and dual-camera super-resolution.

[0012] A second aspect of this application provides an image restoration method, the method comprising: Obtain the target input image to be repaired; The target input image is input into a diffusion model for image inpainting, so that the diffusion model iteratively inpaints the target input image based on a reverse denoising process and outputs the inpainted image; wherein the diffusion model for image inpainting is trained based on the data consistency training method for the diffusion model for image inpainting.

[0013] A third aspect of this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the data consistency training method for the diffusion model for image inpainting, and / or to implement the image inpainting method.

[0014] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the data consistency training method for the diffusion model for image inpainting, and / or implements the image inpainting method.

[0015] A fifth aspect of this application provides a computer program product comprising a computer program that, when executed by a processor, implements the data consistency training method for the diffusion model for image inpainting, and / or implements the image inpainting method.

[0016] This application provides a data consistency training method for a diffusion model for image inpainting. The method includes: during the data consistency training process for the diffusion model for image inpainting, acquiring the training input data of the diffusion model at step t; wherein the training input data is intermediate image data obtained after iterating to step t according to the inverse denoising process of the diffusion model, using an input image to be inpainted as the starting point; T represents the total number of denoising iterations of the diffusion model, and t represents an integer from T to 1; inputting the training input data at step t and the information representing step t into the core network of the diffusion model to obtain the prediction result output by the core network; calculating a loss function based on the prediction result and the supervision data corresponding to step t, and updating the parameters of the core network according to the loss function; wherein the intermediate image data obtained by the inverse denoising process is used as the training input data, so that the training process optimizes the cumulative error generated by the diffusion model during multi-step denoising iterations in the testing phase while optimizing the single-step denoising error. By using a reverse process to generate input data during the training phase and aligning the training and testing processes, the problem of inconsistent training and testing input data in traditional models can be solved, allowing the model to directly optimize accumulated errors and improve restoration accuracy from the root. Through dynamic programming and intermediate result caching mechanisms, the computational overhead and memory usage during the training phase can be reduced, achieving a balance between performance and efficiency of the image restoration model. At the same time, the method of this application can be applied to multiple image restoration tasks such as single image super-resolution, image denoising, image deraining, and dual-camera super-resolution, achieving multi-task adaptation.

[0017] Additional advantages, objectives, and features of this application will be set forth in part in the description which follows, and will in part become apparent to those skilled in the art upon review of the following description, or may be learned by practice of the application. The objectives and other advantages of this application can be realized and obtained by means of the structures specifically pointed out in the specification and drawings.

[0018] Those skilled in the art will understand that the purposes and advantages that can be achieved with this application are not limited to those specifically described above, and that the above and other purposes that this application can achieve will be more clearly understood from the following detailed description. Attached Figure Description

[0019] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, do not constitute a limitation thereof. The components in the drawings are not drawn to scale but are merely for illustrating the principles of this application. For ease of illustration and description of certain parts of this application, corresponding portions in the drawings may be enlarged, i.e., may appear larger relative to other components in an exemplary device actually manufactured according to this application. In the drawings: Figure 1 This is a flowchart illustrating the data consistency training method for a diffusion model used for image inpainting according to one embodiment of this application.

[0020] Figure 2 This is a schematic diagram showing the results of quantitative analysis and modeling of the cumulative error of the traditional diffusion model in an application example of this application.

[0021] Figure 3 This is a schematic diagram of the iterative process of the traditional method model in an application example of this application.

[0022] Figure 4 This is a schematic diagram of the traditional method for training loss in an application example of this application.

[0023] Figure 5 This is a flowchart illustrating the training loss of the data consistency training method in an application example of this application.

[0024] Figure 6 This is a flowchart illustrating the data consistency training method in an application example of this application.

[0025] Figure 7 This is a flowchart illustrating the efficient version of the data consistency training method in an application example of this application.

[0026] Figure 8 This is a flowchart illustrating the image restoration method in an application example of this application. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and their descriptions are used to explain this application, but are not intended to limit it.

[0028] It should also be noted that, in order to avoid obscuring this application with unnecessary details, only the structures and / or processing steps closely related to the solution according to this application are shown in the accompanying drawings, while other details that are not closely related to this application are omitted.

[0029] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0030] It should also be noted that, unless otherwise specified, the term "connection" in this article can refer not only to a direct connection, but also to an indirect connection involving an intermediary.

[0031] In the following description, embodiments of the present application will be illustrated with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar parts, or the same or similar steps.

[0032] Existing DDM-based image inpainting model implementations include efficient denoising diffusion model backbone networks (ResidualShift Network, ResShift), diffusion-based plug-and-play plugin models (Diff-Plugin), diffusion-based unsupervised image restoration models (DiffUIR), and iterative optimization frameworks. ResShift, an efficient backbone network optimized based on a latent diffusion model, reduces computational costs by decreasing the number of iterations, making it suitable for image super-resolution tasks. Diff-Plugin and DiffUIR are diffusion model variants designed for image inpainting, improving inpainting performance by adjusting the network structure or adding preprocessing and post-processing modules.

[0033] However, existing DDM-type image inpainting models all follow a traditional training framework, in which iterative input data is generated through a forward process (adding noise step by step) during the training phase. The core network learns to minimize single-round module errors; during the testing phase, input data is generated through a reverse process (multi-step iterative denoising). The image quality is gradually optimized. Due to the inconsistent training-testing process of existing DDM-type image inpainting models, the cumulative error is out of control, the computational cost is high, and the generalization ability is limited.

[0034] Therefore, in order to solve the problems of high cumulative error, large computational overhead and low adaptability of existing training methods for image inpainting, this application provides a data consistency training method for diffusion models for image inpainting, an image inpainting method, an electronic device, a computer-readable storage medium and a computer program product, respectively, to correct the cumulative error, reduce the computational overhead and memory usage during the training phase, improve the inpainting accuracy from the root and achieve multi-task adaptability.

[0035] The following examples will provide a detailed description.

[0036] Based on this, embodiments of this application provide a data consistency training method for a diffusion model used for image inpainting, which can be executed by a data consistency training device for a diffusion model used for image inpainting. See [link to relevant documentation]. Figure 1 The method specifically includes the following: Step 100: During the data consistency training process for the diffusion model used for image inpainting, the training input data of the diffusion model at step t is obtained; wherein, the training input data is intermediate image data obtained after iterating to step t according to the inverse denoising process of the diffusion model, with an input image to be inpainted as the starting point; T represents the total number of denoising iterations of the diffusion model, and t represents an integer from T to 1.

[0037] Specifically, to ensure that the input data during the training phase is completely consistent with that during the testing phase, and to allow the model to access intermediate data containing error propagation during training, rather than simply learning denoising from ideal noisy data, this is achieved by adjusting the training input... Generated step-by-step from the forward process Replace with multi-step generation of the reverse process By actively introducing the error propagation characteristics during the testing phase, the accumulated error can be directly optimized by the training loss.

[0038] Understandably, the training input for traditional DDM originates from the forward process. Where q represents a single-step noise addition operation, This indicates a high-quality image (i.e., a restored image). This represents an intermediate state image. Its generation logic is based on the truth value. Adding single-step controllable noise essentially involves generating ideal, error-free noise data. The input for the testing phase originates from the reverse process, whose generation logic is derived from low-quality images. Obtained through core network denoising The process inevitably carries over the accumulated error from the previous step. It is real, intermediate data that includes error propagation.

[0039] Step 200: Input the training input data of step t and the information representing step t into the core network of the diffusion model to obtain the prediction result output by the core network.

[0040] It should also be noted that, obey Distribution, representing from arrive The process involves a multi-step reverse denoising step. Specific implementation details are provided in subsequent embodiments.

[0041] in, This represents the image data at iteration step i. The image represents the iteration step (i+1). Let p represent the image data at iteration step T, and p represent the distribution. Indicates model parameters, This indicates the distribution of data generated by the model. Indicates known In this case, the model predicts The distribution of the equations is an iterative process, starting from the known... According to the model, The final predicted output is obtained. .

[0042] Step 300: Calculate the loss function based on the prediction result and the supervision data corresponding to step t, and update the parameters of the core network according to the loss function; wherein, the intermediate image data obtained by the reverse denoising process is used as the training input data, so that the training process optimizes the single-step denoising error while optimizing the cumulative error generated by the diffusion model during multi-step denoising iteration in the testing phase.

[0043] Specifically, after the input data is replaced, the mathematical relationship between module error, input cumulative error, and total cumulative error is fundamentally reconstructed, ultimately achieving equivalence between cumulative error and module error, allowing the training loss to simultaneously optimize both types of errors. Because the training input becomes... The benchmark for calculating module error is based on the ideal input. Turn into real input New module error Defined as: Where t represents the iteration step, This represents the input image data. This represents the supervision signal for each iteration step during the training phase. Represents model parameters. Error compared to traditional modular arithmetic. compared to, It not only includes the denoising error of the core network on the current step's data, but also implicitly includes the impact of error propagation from the previous step, because It carries its own input cumulative error ,therefore Essentially, it is a comprehensive reflection of the current denoising error and the previous propagation error. The definition is reverse process input Compared to ideal input The difference, and in the method of this embodiment, That is, the training input is the input of the reverse process, and at this time the ideal input With training input Completely equivalent: This key change completely eliminates the additional source of input cumulative error, making the calculation of the total cumulative error depend solely on the new module error. Final cumulative error Will Substituting into the traditional cumulative error formula: achievable Therefore, the total cumulative error is completely equivalent to the error of the new module; optimizing the error of the new module is equivalent to optimizing the total cumulative error. The loss function can adopt the MSE loss framework of the traditional diffusion model, but the input data is replaced with... The formula is: In the formula, This represents the weight coefficient for each iteration step. Because... This loss function optimizes both the module error and the cumulative error simultaneously.

[0044] It's important to note that this replacement introduces variational noise perturbations unique to the testing phase into the training phase. Specifically, the tiny errors generated by each denoising step in the reverse engineering process are iteratively propagated to subsequent steps, resulting in an error distribution consistent with that of the testing phase. This proactive introduction of realistic error scenarios compensates for the shortcomings of traditional training methods that are detached from actual testing, providing a data foundation for optimizing accumulated errors.

[0045] As described above, the data consistency training method for a diffusion model for image restoration provided in this application solves the problem of inconsistent training and testing of input data in traditional models by generating input data through a reverse process during the training phase and aligning the training and testing processes. This allows the model to directly optimize accumulated errors and improve restoration accuracy from the root. By using dynamic programming and intermediate result caching mechanisms, the computational overhead and memory usage during the training phase can be reduced, achieving a balance between performance and efficiency of the image restoration model. Furthermore, the method of this application can be applied to multiple image restoration tasks such as single-image super-resolution, image denoising, image deraining, and dual-camera super-resolution, achieving multi-task adaptation.

[0046] To further optimize the accumulated error and improve the restoration accuracy, in a data consistency training method for a diffusion model for image restoration provided in this application embodiment, step 100, which involves obtaining the training input data of the diffusion model at step t, specifically includes the following: Step 110: Associate the total number of denoising iterations T with the order of training rounds, and the training input data obtained in the k-th training round corresponds to the t-th step; where t = T - k + 1, and k represents an integer from 1 to T.

[0047] Step 120: In the first training round, i.e., at t=T, the input image to be repaired is used as the training input data.

[0048] Step 130: In the k-th training round, and k≥2, load and reuse the intermediate image data generated and cached in the (k-1)-th training round as the training input data for the current t-th step.

[0049] Specifically, the iteration steps are associated with the training rounds, and redundant computations are reduced by caching and reusing intermediate results. This significantly reduces training resources and time while sacrificing a small amount of disk storage. The iteration steps of the diffusion model range from 1 to T, corresponding to the restoration process from high-quality intermediate data close to the truth to the input image to be restored. The training rounds proceed in the order t from T to 1: the first round processes t=T, the second round processes t=T-1, ..., the Tth round processes t=1.

[0050] To further optimize the accumulated error and improve the restoration accuracy, the data consistency training method for the diffusion model for image restoration provided in this application embodiment includes the following content after step 130: Step 140: After completing the parameter update for the kth training round, use the updated core network to process the training input data for the current tth step, generate intermediate image data for the (t-1)th step and cache it for use in the (k+1)th training round.

[0051] Specifically, in each round, the model only learns the denoised mapping of the current round k at iteration step t, and the input data... The cache from the previous k-1 iteration step (t+1 step) can be directly reused. As a result, there is no need to calculate the reverse process from scratch.

[0052] To further optimize the accumulated error and improve the repair accuracy, in the data consistency training method for a diffusion model for image repair provided in this application embodiment, the supervision data corresponding to the t-th step represents the preset ground image data corresponding to the input image to be repaired, which serves as the optimization target for the prediction result.

[0053] Specifically, the supervision data is the same as the repaired image corresponding to the input image to be repaired, in order to train the model.

[0054] To further optimize the accumulated error and improve the restoration accuracy, in the data consistency training method for a diffusion model for image restoration provided in this application embodiment, the loss function specifically includes the following: At least one of mean squared error loss, KL divergence loss, and perception loss.

[0055] Specifically, the loss function can be an indicator that measures the difference between two probability distributions (Kullback-Leibler Divergence, KL divergence), perceptual loss, or mean-square error (MSE) loss function. By maintaining the core logic of optimizing the cumulative error based on the reverse process input, the repair accuracy can be improved.

[0056] To further optimize the accumulated error and improve the restoration accuracy, in the data consistency training method for a diffusion model for image restoration provided in this application embodiment, the core network specifically includes the following: Denoising networks based on U-Net, latent diffusion models, residual offset networks, or Transformer architectures.

[0057] Specifically, the core network can use other DDM variants (such as Latent Diffusion Model (LDM), Denoising Diffusion Probabilistic Model (DDPM), Diffusion Transformer for Super-Resolution (DiT4SR), etc.), ResShift, etc. Only the number of iterations T and hyperparameter ηt during the training process need to be adjusted to achieve data-consistent training; To further optimize the accumulated error and improve the repair accuracy, in the data consistency training method for the diffusion model for image repair provided in this application embodiment, the image repair specifically includes at least one of the following: single image super-resolution, image denoising, image deraining, and dual-camera super-resolution.

[0058] Specifically, it can adapt to multiple image restoration tasks such as single-image super-resolution, image denoising, image deraining, and dual-camera super-resolution without modifying the network structure.

[0059] To further illustrate the above embodiments, this application also provides a specific application example of a data consistency training method for a diffusion model used for image inpainting, which specifically includes the following: 1. Basic Principles Quantitative analysis and modeling of cumulative error in traditional diffusion models, such as Figure 2 As shown. Figure 2 The x-axis represents the number of iterations in the DDM, and the y-axis represents the model predictions. and the real The smaller the KL divergence, the better. The dashed line represents module error, which refers to the processing error of the model within one iteration step to ensure that the model input data is error-free. The solid line represents the cumulative error of the model input data, which is the error obtained from the last iteration step T. , The error is used as input data for the model. As you can see, the module error can be kept very low during training, but the cumulative error gets bigger and bigger.

[0060] This application example is the first to clearly define the errors of traditional diffusion models in image inpainting as either module error or cumulative error, and quantifies the relationship between the two using mathematical formulas. Module error The difference between a single-step network output and the true value is defined as: Where D represents the KL divergence, The model parameters are represented by 'unet' (the commonly used model is 'unet'), and 't' represents the iteration step. This represents the supervision signal at iteration step t during the training phase, analogous to traditional models. t and This is a concept related to diffusion models. The difference between diffusion models and general models is that, in addition to the data input, a diffusion model also inputs a value 't' representing the iteration step, thus informing the model which iteration step the current data is training on. Cumulative error. The total error from the initial iteration to iteration step t is measured by the combined influence of module error and the accumulated error from the previous step, and is expressed by the following formula: And input cumulative error ( (Input for the reverse process). Experimental observations revealed that traditional training only optimizes... (Training phase) (Low), but in the testing phase With continuous growth through iteration (e.g.) Figure 2 As shown, the traditional diffusion model (The error is much higher than the module error), and this modeling provides a theoretical basis for subsequent design.

[0061] In addition, such as Figure 3 As shown, This represents the image data (low-quality input) for iteration step T. This refers to the final generated prediction result. The iteration of the model during testing refers to... arrive ...finally The process. During training, the model is based on... And t to predict The process is as follows Figure 3 middle As shown, and This represents the forward process. Traditional methods... Through the forward process Therefore, the application example in this application is based on the reverse process. get.

[0062] 2. Detailed technical solution (1) Data-Consistent Training (DCT) Method To address the fundamental problem of inconsistent training and testing inputs in traditional diffusion models, the DCT method is proposed. Its core innovation lies in aligning the input data generation logic between the training and testing phases, by generating the training input step-by-step from the forward process. Replace with multi-step generation of the reverse process By proactively introducing the error propagation characteristics of the testing phase, the accumulated error can be directly optimized by the training loss, thereby bridging the performance gap between training and testing.

[0063] 1) Consistent replacement of input data: Transmission from ideal noise to real error Traditional DDM training input Originating from the forward process Its generation logic is based on the truth value Adding single-step controllable noise essentially results in ideal, error-free noise data. The input during the testing phase... Originating from the reverse process Its generation logic is based on low-quality images. Obtained through core network denoising The process inevitably carries over the accumulated error from the previous step. It is real, intermediate data that includes error propagation.

[0064] The first step of the DCT method is to input the training phase data... from Replace with In traditional methods: Where q represents a single-step noise addition operation, and These represent high-quality images and intermediate-state images, respectively. In the DCT method... obey Distribution, representing from arrive The multi-step reverse denoising process aims to ensure that the input data during the training phase is completely consistent with that during the testing phase, so that the model can access intermediate data containing error propagation during training, rather than just learning the denoising of ideal noisy data.

[0065] This replacement introduces variational noise perturbations unique to the testing phase into the training phase. These perturbations are the tiny errors generated by denoising at each step of the inverse process, which are iteratively passed to subsequent steps, forming an error distribution consistent with the testing phase. This design, which proactively introduces real-world error scenarios, completely overcomes the shortcomings of traditional training that is detached from actual testing, providing a data foundation for optimizing accumulated errors.

[0066] Specific process 2) Reconstruction of error relationships: enabling direct optimization of accumulated errors. After the input data is replaced, the mathematical relationship between module error, input cumulative error and total cumulative error is fundamentally reconstructed, ultimately achieving equivalence between cumulative error and module error, so that the training loss can simultaneously optimize the two types of errors.

[0067] New module error ( ): Because the training input becomes The benchmark for calculating module error is based on the ideal input. Turn into real input Defined as: Error compared to traditional modules compared to, It not only includes the denoising error of the core network on the current step's data, but also implicitly includes the impact of error propagation from the previous step, because Carrying itself ,therefore Essentially, it reflects the combined effect of the current denoising error and the previous propagation error. For example... Figure 4 As shown, the input data for traditional training models yes The noise distribution obtained by single-step noise addition is a pre-assumed noise distribution that follows the Markov assumption. In the DCT method, such as... Figure 5 As shown, the model input data yes Consistent with the testing phase, it carries accumulated error. t represents the iteration step and t∈[1,T] (representing an integer from 1 to T). This represents the supervision signal for each iteration step during the training phase. Indicates model parameters.

[0068] Elimination of cumulative input error ( Input cumulative error The definition is reverse process input Compared to ideal input The difference, and in the DCT method, That is, the training input is the input of the reverse process, and at this time the ideal input With training input They are completely equivalent, therefore: This key change completely eliminates the additional source of error, the input cumulative error, so that the calculation of the total cumulative error depends only on the new module error.

[0069] Final cumulative error ( ):Will Substituting into the traditional cumulative error formula, we get: At this point, the total cumulative error and the new module error are completely equivalent; optimizing the new module error is equivalent to optimizing the total cumulative error. This reconstruction of the mathematical relationship is the core breakthrough of the DCT method: it transforms the cumulative error, which is inaccessible in traditional training, into a module error that can be directly optimized through the loss function, thus aligning the error optimization objective with the testing requirements.

[0070] 3) Compatibility of training loss design and methods Based on the above error reconstruction, the training loss function of the DCT method can follow the MSE loss framework of the traditional diffusion model, but the input data is replaced with... The formula is: In the formula, These are the weight coefficients for each iteration step. Because... This loss function optimizes both module error and cumulative error simultaneously, achieving a dual optimization effect.

[0071] (2) Efficient Data-Consistent Training (Effi-DCT) method based on dynamic programming strategy Although the DCT method can effectively suppress accumulated error, it suffers from limitations in training input... It needs to be generated in multiple steps through a reverse process (from) arrive Each training round requires repeated computation of intermediate data from all iterations, leading to a significant increase in memory usage and time costs during the training phase, making it difficult to meet the requirements of engineering deployment. The DCT method, for example... Figure 6 As shown, t represents the iteration step and t∈[1,T] (representing an integer from 1 to T). , , Different input images are represented by k, which is used to distinguish different intermediate data during training and is a positive integer between 0 and T (its meaning is the same as t, but k=0 represents the final generated result). The diagram uses T=3 as an example. The red arrow represents the calculation of the generated input data, and the green arrow represents the inference of the diffusion model. The specific process is as follows: ① First round: Assume there are 3 data points: a, b, and c. The iteration step t in each round is randomly generated. When t=1, the DCT method obtains the input data for the model. ,from get , and then from Finally obtained (The red arrows in the diagram represent redundant calculations), resulting in Then, the model was used to predict and obtain The same applies to t=2 and / or t=3.

[0072] ② Second round: Randomly generate 3 data points corresponding to iteration steps t, and perform the same operation process as in the first round; ③ Subsequent rounds: Randomly generate 3 data points corresponding to iteration steps t, and perform the same operation process as in the first round; Repeat steps ①-③ until the loss level stabilizes, the model converges, and training ends.

[0073] To address this efficiency bottleneck, the Effi-DCT method is designed based on dynamic programming. Its core principle is to associate the iteration step t with the training rounds and reduce redundant computations by caching and reusing intermediate results. Figure 6 The section marked with the red arrow demonstrates how training resources and time are significantly reduced while sacrificing a small amount of disk storage.

[0074] 1) The core logic of dynamic programming: the relationship between iteration steps and rounds. The essence of dynamic programming is to decompose complex problems into overlapping subproblems and avoid redundant computation by storing the results of the subproblems. In the DCT method, generation... This core task has significant overlap in its sub-problems and requires reliance on The calculation results, and This has already been calculated in previous rounds. Based on this, the iteration step t is strongly correlated with the training round, so that each round focuses only on the input generation and model training of one iteration step. The specific logic is as follows: The iteration step range of the diffusion model is t∈[1,T] (representing an integer from 1 to T), corresponding to the restoration process from high-quality intermediate data close to the ground truth to the input image to be restored; the training rounds proceed in the order of t from T to 1: the first round processes t=T, the second round processes t=T-1, ..., the Tth round processes t=1; in each round, the model only learns the denoising mapping of the current iteration step t, and the input data... The cache from the previous round (t+1) can be directly reused. As a result, there is no need to calculate the reverse process from scratch.

[0075] The key advantage of this associated design is that it breaks down the high-overhead task of calculating all intermediate data of all iteration steps at once into a lightweight task of calculating data of a single iteration step round by round, and the calculation of each step is based on the cached result of the previous step, thus completely eliminating redundant calculations.

[0076] 2) Specific implementation of the Effi-DCT method Effi-DCT method such as Figure 7 As shown, t represents the iteration step and t∈[1,T] (representing an integer from 1 to T). , , Different input images are represented by k, which is used to distinguish different intermediate data during training and is a positive integer between 0 and T (its meaning is the same as t, but k=0 represents the final generated result). The diagram uses T=3 as an example, and the green arrow represents the inference of the diffusion model. The specific process is as follows: ① First round: Iteration steps for all data , Low-quality input images can be used as training input without any computation; ②After the first round: The training network has mastered the knowledge from... arrive The mapping of all data Cache to disk, such as Figure 7 Data in the cached area; ③ Second round: Iteration steps for all data Loaded directly from the cache area As input; ④ Subsequent rounds: The iteration step t is decreased by 1 each time, and the input is read from the previous cache until t=1.

[0077] ⑤ Repeat steps ①-④ until the loss level stabilizes, the model converges, and training ends.

[0078] 3. Experimental verification (1) Objective results of the denoising task (as shown in Table 1) Table 1 Objective Results of the Noise Removal Task Table 1 shows the comparison results between the proposed method and 13 mainstream image denoising models on five metrics: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), Learned Perceptual Image Patch Similarity (LPIPS), Fréchet Inception Distance (FID), and CLIP-based Quality Assessment (CLIPQA). PSNR and SSIM are used to evaluate the fidelity of the denoised image (higher values ​​are better), LPIPS and FID measure the perceptual difference from the real image (lower values ​​are better), and CLIPQA reflects the semantic quality of the image (higher values ​​are better).The 13 mainstream image denoising models include: Domain-Adaptive CLIP (DA-CLIP), a domain-adaptive method based on contrastive language-image pre-training; Swin Transformer-based Image Restoration (SwinIR); Adaptive Image Restoration Network (AirNet); Diffusion-based Plug-and-Play Plugin (DiffPlugin); Diffusion-based Unsupervised Image Restoration (DiffUIR); Prompt-based Image Restoration (PromptIR); Efficient Transformer for Image Restoration (Restormer); Multi-scale Augmentation foreXtreme Image Restoration (MAXIM); and Blind to Unblind Image Restoration. Image restoration models include Blind2Unblind, Kernel Basis Network (KBNet), Image Processing Transformer (IPT), Structure-Aware Denoising Network (SADNet), and Residual Shift Network (ResShift).

[0079] The data shows that the DCT model proposed in this application demonstrates the best performance in core fidelity metrics, achieving a PSNR of 32.25 dB and an SSIM of 0.8989. These values ​​are 0.44 dB higher than the second-best model, MAXIM (30.81 dB, 0.9030), and slightly lower than MAXIM (0.0041), respectively, indicating a leading overall fidelity level. In terms of perceptual quality, the DCT model's LPIPS value of 0.0812 is the lowest among all comparison methods, indicating that its denoising results are visually closest to the real image. In terms of FID, the DCT model ranks second with a score of 61.82, second only to the efficient version Effi-DCT (59.17), demonstrating stronger image distribution consistency. In the CLIPQA metric, the DCT model ranks second with a score of 0.2090, only 0.0017 lower than KBNet (0.2073), demonstrating excellent semantic quality performance. Furthermore, the proposed Effi-DCT in the application examples of this application maintains high performance (PSNR 30.87dB, SSIM 0.8887) while achieving an FID score of 59.17, the best among all models, verifying its good balance between efficiency and performance. Overall, the proposed method achieves comprehensive improvements in fidelity, perceptual quality, and semantic consistency in denoising tasks, outperforming classic models such as DA-CLIP, SwinIR, and Restormer.

[0080] (2) Objective results of the rain removal task (as shown in Table 2) Table 2 Objective Results of Rain Removal Tasks Table 2 presents an objective performance comparison of 12 mainstream rain removal models and the method proposed in the application example of this application in the rain removal task. The 12 mainstream rain removal models include: DA-CLIP, SwinIR, AirNet, DiffPlugin, DiffUIR, PromptIR, Restormer, MAXIM, Weakly Guided Wavelet Super-resolution (WGWS), KBNet, IPT, and ResShift.

[0081] Both achieved the best results, with a PSNR of 34.22 dB and an SSIM of 0.9359, representing improvements of 0.31 dB and 0.0098 respectively compared to the second-best model, DA-CLIP (33.91 dB and 0.9261). This indicates that it can preserve image detail to the greatest extent while removing rain streaks. In terms of perceptual quality, the DCT model has an LPIPS value of 0.0747, the lowest among all compared methods, far lower than DA-CLIP (0.0942) and Restorm. The accuracy of the perceptual index (r) is 0.1289, indicating that the deraining result is visually more natural. In terms of FID (Fairness Identification), Effi-DCT ranks first with 79.84, and the DCT model ranks fourth with 98.85, outperforming most of the comparison models. In the CLIPQA (Content-Based Quality Assessment) index, ResShift ranks first with 0.2508, and the DCT model ranks second with 0.2276, significantly higher than DA-CLIP (0.1239) and Restormer (0.1297), demonstrating outstanding semantic quality. Effi-DCT's PSNR is 32.75dB, SSIM is 0.9179, and LPIPS is 0.0910, ranking second among all comparison models, showcasing a balance between efficiency and high performance. Overall, the method proposed in this application demonstrates strong competitiveness in the deraining task, effectively removing rain streaks while maintaining image fidelity and perceptual quality, outperforming existing mainstream deraining models.

[0082] (3) Objective results of the SISR task (CameraFusion dataset, as shown in Table 3) Table 3 Objective results of the SISR task (CameraFusion dataset) Table 3 shows the objective performance comparison of 15 mainstream single-image super-resolution (SISR) models and the application example method in this application on the CameraFusion dataset. PSNR and SSIM reflect the reconstruction fidelity of the super-resolution image, LPIPS and FID measure perceptual similarity, and CLIPQA evaluates semantic quality. The 15 mainstream rain removal models include: DA-CLIP, SwinIR, AirNet, DiffPlugin, DiffUIR, PromptIR, Restormer, MAXIM, Real-World Super-Resolution via Enhanced GAN (RealESRGAN), Diffusion-based Image Restoration (DiffIR), Hierarchical Attention Transformer (HAT), Semantic Enhancement for Super-Resolution (SeeSR), Stable Diffusion-based Super-Resolution (StableSR), Super-Resolution Transformer (SRFormer), and ResShift.

[0083] The results show that the DCT model proposed in this application achieves the best performance in all three metrics: PSNR (31.41 dB), SSIM (0.8391), and LPIPS (0.0427). Specifically, the PSNR is 0.02 dB higher than the second-best model, Effi-DCT (31.39 dB), and the SSIM is 0.0123 higher than StableSR (0.8268). Restormer's LPIPS is 0.0269, the lowest among all models, while the DCT model ranks second with 0.0427. In terms of perceptual quality, the DCT model ranks first with a score of 27.64, second only to Restormer, but lower than SeeSR (28.31) and ResShift (28.58), indicating that its super-resolution results have the smallest distribution difference from the real image. In the CLIPQA index, StableSR ranks first with a score of 0.1181, and the DCT model ranks second with a score of 0.1006, higher than RealESRGAN (0.0921) and AirNet (0.0889), demonstrating leading semantic quality. Effi-DCT has a PSNR of 31.39dB, an SSIM of 0.8148, and an FID of 29.62, ranking second among all compared models, only slightly lower than the DCT model, verifying its efficiency. In summary, the proposed method in the application examples of this application performs excellently in terms of fidelity, perceptual quality, and semantic consistency on the SISR task of the CameraFusion dataset, outperforming mainstream super-resolution models such as DiffIR, HAT, and StableSR, demonstrating strong practical application value.

[0084] (4) Experimental conclusions The application examples of this application can solve the problem of inconsistent training and testing of DDM model input data, and significantly improve the repair accuracy: compared with traditional diffusion model training methods, the PSNR of this invention is improved by 4.13dB (single image super-resolution), 1.48dB (denoising), 5.57dB (rain removal), and 2.45dB (dual-camera super-resolution) on four major tasks, respectively, and all quality indicators (SSIM / LPIPS / FID) are superior to existing mainstream methods (such as Restormer, MAXIM, SwinIR, ResShift). Computational overhead is controllable: The efficient DCT solution reduces training memory usage by 52.5% and training time by 68.9% while maintaining the same performance. The latency and memory usage during the testing phase are consistent with traditional models, and it has the capability for engineering deployment. It has strong versatility for multiple tasks: there is no need to modify the network structure for specific tasks. It can be adapted to four major scenarios: single image super-resolution, denoising, rain removal, and dual-camera super-resolution by simply adjusting the input data. It can also be transferred to different diffusion model backbone networks (such as LDM and ResShift), all of which can bring a PSNR improvement of 1.5-3dB. Excellent robustness across multiple datasets: It performs stably on both real datasets (such as RealSR) and synthetic datasets (such as CameraFusion), avoiding the performance degradation caused by the difference in training and testing data distribution in traditional models.

[0085] Among them, Effi-DCT's caching mechanism can use batch caching instead of full caching, and dynamically adjust the number of caching iterations based on the size of the hardware memory, which can still reduce computational overhead in scenarios with limited storage resources; the number of iterations T can be adjusted according to task requirements (such as T=10 or T=20), and hyperparameters such as acceleration factor and learning rate scheduling strategy can be adjusted to adapt to different hardware environments and datasets.

[0086] Based on the embodiments and / or application examples of the data consistency training method for the diffusion model used for image inpainting described above, this application also provides an embodiment of an image inpainting method, see [link to embodiment]. Figure 8 The image restoration method specifically includes the following: Step 001: Obtain the target input image to be repaired; Step 002: Input the target input image into the diffusion model for image inpainting, so that the diffusion model iteratively inpaints the target input image based on the reverse denoising process and outputs the inpainted image; wherein, the diffusion model for image inpainting is trained based on the data consistency training method of the diffusion model for image inpainting.

[0087] It is understood that the data consistency training method for the diffusion model for image restoration mentioned in the embodiments of the image restoration method provided in this application can be implemented based on the processing flow of the embodiment of the data consistency training method for the diffusion model for image restoration in the above embodiments. The content will not be repeated here, but can be referred to the detailed description of the embodiment of the data consistency training method for the diffusion model for image restoration.

[0088] As can be seen from the above description, the image restoration method provided in this application embodiment can be applied to multiple image restoration tasks such as single image super-resolution, image denoising, image deraining, and dual-camera super-resolution, achieving multi-task adaptation.

[0089] This application also provides an electronic device, which may include a processor, a memory, a receiver, and a transmitter. The processor is used to execute the data consistency training method for the diffusion model for image inpainting mentioned in the above embodiments and / or the image inpainting method. The processor and the memory can be connected via a bus or other means, taking a bus connection as an example. The receiver can be connected to the processor and the memory via wired or wireless means.

[0090] The processor can be a central processing unit (CPU). The processor can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations of the above types of chips.

[0091] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the data consistency training method for the diffusion model for image inpainting and / or the program instructions / modules corresponding to the image inpainting method in the embodiments of this application. The processor executes various functional applications and data processing by running the non-transitory software programs, instructions, and modules stored in the memory, thereby implementing the data consistency training method for the diffusion model for image inpainting and / or the image inpainting method in the above method embodiments.

[0092] The memory may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created by the processor, etc. Furthermore, the memory may include high-speed random access memory and non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include memory remotely located relative to the processor, which can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0093] The one or more modules are stored in the memory, and when executed by the processor, they execute the data consistency training method for the diffusion model for image inpainting and / or the image inpainting method in the embodiment.

[0094] In some embodiments of this application, the user equipment may include a processor, a memory, and a transceiver unit. The transceiver unit may include a receiver and a transmitter. The processor, memory, receiver, and transmitter may be connected via a bus system. The memory is used to store computer instructions, and the processor is used to execute the computer instructions stored in the memory to control the transceiver unit to send and receive signals.

[0095] As one implementation method, the functions of the receiver and transmitter in this application can be implemented by transceiver circuits or dedicated transceiver chips, and the processor can be implemented by dedicated processing chips, processing circuits or general-purpose chips.

[0096] As another implementation approach, the server provided in this application embodiment can be implemented using a general-purpose computer. That is, the program code implementing the processor, receiver, and transmitter functions is stored in memory, and the general-purpose processor implements the processor, receiver, and transmitter functions by executing the code in memory.

[0097] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the aforementioned data consistency training method for a diffusion model for image inpainting and / or the steps of the image inpainting method. The computer-readable storage medium may be a tangible storage medium, such as random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, register, floppy disk, hard disk, removable storage disk, CD-ROM, or any other form of storage medium known in the art.

[0098] This application also provides a computer program product, specifically comprising a computer program that, when executed by a processor, implements the data consistency training method for the diffusion model for image inpainting mentioned in the foregoing embodiments and / or the steps of the image inpainting method.

[0099] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed in the application examples of this application can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. The programs or code segments can be stored in a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave.

[0100] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0101] In this application, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0102] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to the embodiments of this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A data consistency training method for a diffusion model used for image inpainting, characterized in that, The method includes: During the data consistency training process for a diffusion model used for image inpainting, the training input data of the diffusion model at step t is obtained; wherein, the training input data is intermediate image data obtained after iterating to step t according to the inverse denoising process of the diffusion model, with an input image to be inpainted as the starting point; T represents the total number of denoising iterations of the diffusion model, and t represents an integer from T to 1. The training input data at step t and the information representing step t are input into the core network of the diffusion model to obtain the prediction result output by the core network. The loss function is calculated based on the prediction result and the supervision data corresponding to step t, and the parameters of the core network are updated according to the loss function; wherein, the intermediate image data obtained by the reverse denoising process is used as the training input data, so that the training process optimizes the single-step denoising error and the cumulative error generated by the diffusion model during multi-step denoising iteration in the testing phase.

2. The data consistency training method for the diffusion model for image inpainting according to claim 1, characterized in that, The step of obtaining the training input data of the diffusion model at step t includes: The total number of denoising iterations T is associated with the order of training rounds, and the training input data obtained in the k-th training round corresponds to the t-th step; where t = T - k + 1, and k represents an integer from 1 to T; In the first training round, i.e., at t=T, the input image to be repaired is used as the training input data; In the k-th training round, and when k≥2, the intermediate image data generated and cached in the (k-1)-th training round is loaded and reused as the training input data for the current t-th step.

3. The data consistency training method for the diffusion model for image inpainting according to claim 2, characterized in that, Also includes: After completing the parameter update for the kth training round, the updated core network is used to process the training input data for the current t-th step, generate intermediate image data for the (t-1)th step, and cache it for use in the (k+1)th training round.

4. The data consistency training method for the diffusion model for image inpainting according to claim 1, characterized in that, The supervision data corresponding to step t represents the preset ground truth image data corresponding to the input image to be repaired, which serves as the optimization target for the prediction result.

5. The data consistency training method for the diffusion model for image inpainting according to claim 1, characterized in that, The loss function includes at least one of the following: mean squared error loss, KL divergence loss, and perceptual loss.

6. The data consistency training method for the diffusion model for image inpainting according to claim 1, characterized in that, The core network includes: a denoising network based on U-Net, a latent diffusion model, a residual offset network, or a Transformer architecture.

7. The data consistency training method for the diffusion model for image inpainting according to claim 1, characterized in that, The image restoration includes at least one of the following: single-image super-resolution, image denoising, image deraining, and dual-camera super-resolution.

8. An image restoration method, characterized in that, The method includes: Obtain the target input image to be repaired; The target input image is input to a diffusion model for image inpainting, so that the diffusion model iteratively inpaints the target input image based on a reverse denoising process and outputs the inpainted image; wherein the diffusion model for image inpainting is trained based on the data consistency training method for the diffusion model for image inpainting according to any one of claims 1 to 7.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the data consistency training method for the diffusion model for image inpainting as described in any one of claims 1 to 7, and / or the image inpainting method as described in claim 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the data consistency training method for a diffusion model for image inpainting as described in any one of claims 1 to 7, and / or the image inpainting method as described in claim 8.

Images