A method and apparatus for fast image generation
By explicitly aligning the noise variance during the diffusion model inversion process, the problem of noise amplitude underestimation is corrected, and high-fidelity image reconstruction with a small number of steps is achieved. This solves the problems of reduced reconstruction quality and semantic drift caused by noise amplitude underestimation in diffusion model inversion, and reduces computational overhead.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- FUDAN UNIVERSITY
- Filing Date
- 2026-05-09
- Publication Date
- 2026-06-09
AI Technical Summary
Existing diffusion model inversion methods suffer from a systematic underestimation of noise amplitude at high-noise time steps, resulting in 'partial denoising' of the inverted trajectory relative to the true forward diffusion trajectory, leading to decreased reconstruction quality and semantic drift. Furthermore, iterative inversion methods have high computational overhead, making it difficult to meet the requirements of training freedom and high efficiency.
By explicitly aligning the variances of the predicted noise and the forward reference noise during the inversion process, and using the noise variance scaling factor for amplitude recalibration, the problem of noise amplitude underestimation is corrected, and high-fidelity reconstruction is achieved with fewer steps, reducing the number of U-Net calls.
Without changing the parameters of the pre-trained diffusion model, it significantly improves the reconstruction quality and semantic fidelity of the inversion, reduces the computational cost, and is suitable for applications such as image enhancement and editing.
Smart Images
Figure CN122176109A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of diffusion model image synthesis and editing technology, specifically relating to a method and apparatus for rapid image generation. Background Technology
[0002] Diffusion models achieve high-quality image generation and editing by progressively injecting noise during the forward noise generation process and progressively denoising during the backward noise generation process. They have achieved significant results in tasks such as text-to-image generation, image inpainting, super-resolution, and personalized generation. To edit realistic images, it is typically necessary to first project the image back into the latent space of the diffusion process. This involves recovering a latent trajectory consistent with the input image content through the inversion stage, allowing subsequent conditional generation or controlled editing on this trajectory to better preserve the original structure and semantics.
[0003] Among existing inversion techniques, DDIM inversion based on deterministic sampling is widely used as a general-purpose inversion tool due to its simplicity, lack of additional training requirements, and compatibility with the original sampling time schedule. This method reverses the DDIM denoising step and uses the output of the noise prediction network on a given latent variable to approximately recover the state of the previous time step, achieving a stepwise inversion from noiseless latent variables to noisy latent variables.
[0004] However, research and practice show that DDIM inversion is prone to reconstruction errors at high-noise time steps. Further analysis reveals a structural asymmetry in its inversion formula: noise prediction during the forward diffusion process depends on the actual forward state. The inversion process uses As input to approximate Noise prediction on the surface. This "use" noise prediction approximation The approximation of "" will accumulate biases in the time dimension, causing the estimation of noise amplitude in the inversion process to be systematically small, i.e., the phenomenon of "under-noising" will occur, thereby inducing a "partial denoising" effect: the latent variables obtained by inversion are not sufficiently disturbed at each step, the overall trajectory is biased to be overly smooth, it is difficult to faithfully restore the real forward diffusion path, and ultimately leads to a decrease in reconstruction quality and semantic drift.
[0005] To mitigate the aforementioned biases, iterative inversion methods such as ReNoise propose performing multiple noise predictions and inversion refinements at each time step. These methods introduce a multi-round iterative mechanism of "adding noise again—inverting again" on top of DDIM inversion, continuously refining the noise prediction at each time step so that the inversion trajectory gradually approximates the forward diffusion trajectory. Empirical evidence shows that the noise statistics (mean and variance) of ReNoise can be divided into two phases during the iteration process: a "landing phase," where the noise variance rapidly increases from a low level to a reasonable range close to the forward distribution; and a "steady phase," where the noise statistics tend to stabilize. Most of the improvement in reconstruction quality occurs during the landing phase; however, this phase also requires a significant amount of U-Net forward computation, leading to a substantial increase in overall inversion costs and hindering efficient deployment in downstream tasks.
[0006] There are two main problems in existing diffusion model inversion: First, DDIM-based inversion methods generally exhibit a systematic underestimation of noise amplitude at high-noise time steps, resulting in "partial denoising" of the inverted trajectory relative to the true forward diffusion trajectory, which leads to a decrease in reconstruction quality, local structural deformation, and semantic drift. Second, although iterative inversion methods (such as ReNoise) can alleviate the above biases by refining noise multiple times, they require multiple calls to the noise prediction network at each time step, resulting in high computational overhead and low inference efficiency, making it difficult to meet the practical application requirements of flexible training, fewer steps, and high efficiency.
[0007] Therefore, how to effectively correct the underestimation of noise amplitude in DDIM inversion while maintaining training freedom, determinism, and a small number of steps, and to obtain reconstruction quality close to or better than ReNoise without relying on multiple iterations, is a technical problem that urgently needs to be solved in this field. Summary of the Invention
[0008] The purpose of this invention is to propose a method and apparatus for rapid image generation, which achieves high-fidelity reconstruction with a small number of steps and significantly reduces the number of U-Net calls during the inversion process.
[0009] The present invention provides a method for fast image generation, comprising: obtaining latent variables from an input image using a pre-trained variational autoencoder. ;
[0010] According to the latent variables Constructing time steps Corresponding forward reference latent variable and the inversion latent variables corresponding to the previous time step ; and based on the aforementioned forward reference latent variable and the inversion latent variables Obtain the predicted noise;
[0011] The variance of the predicted noise is statistically analyzed to obtain a variance scaling factor, and the predicted noise is recalibrated based on the variance scaling factor to obtain the variance-matched noise.
[0012] The time step is obtained using the variance-matched noise. Corresponding inversion latent variables The inversion latent variable trajectory is obtained until time step t > preset inversion time step T. And based on the inversion latent variable trajectory Generate an image.
[0013] As a preferred technical solution, the step of basing the latent variable Constructing time steps Corresponding forward reference latent variable The steps include:
[0014] At time step Sampling Gaussian noise According to the forward diffusion formula Constructing time steps Corresponding forward reference latent variable ,in and These represent the noise-free component coefficient and the noise component coefficient at time step t, respectively.
[0015] As a preferred technical solution, the step of using the forward reference latent variable... and the inversion latent variables The steps of obtaining predicted noise, performing variance statistics on the predicted noise to obtain a variance scaling factor, and recalibrating the predicted noise based on the variance scaling factor to obtain variance-matched noise include:
[0016] The inversion latent variables are respectively With forward reference latent variables Input a preset noise prediction network The predicted noise is obtained. ;in, For noise prediction based on the forward reference trajectory;
[0017] In the channel dimension, variance statistics are performed on the response values of each channel at all positions in both the batch and spatial dimensions to obtain a prediction noise variance tensor of size 1×C×1×1. and reference noise variance tensor ;
[0018] Calculate the noise variance scaling factor based on the difference between the predicted noise variance and the reference noise variance. , To control the hyperparameters of the correction intensity;
[0019] Based on the noise variance scaling factor The predicted noise is recalibrated in amplitude to obtain the variance-matched noise. .
[0020] As a preferred technical solution, the method of obtaining the time step using variance-matched noise is described. Corresponding inversion latent variables The inversion latent variable trajectory is obtained until time step t > preset inversion time step T. The steps include:
[0021] noise after variance matching According to the DDIM inversion update formula Calculate the time step Inversion latent variables and will Increase by 1, let Until time step t > preset inversion time step T, the final inversion latent variables are obtained. As the endpoint of the inversion, As an inversion latent variable trajectory.
[0022] As a preferred technical solution, the variance matching strength control parameter Dynamically adjusted over time steps; or / and;
[0023] The forward distribution used to construct the reference latent variable is in and These represent the noise-free component coefficient and the noise component coefficient at time step t, respectively.
[0024] As a preferred technical solution, the step of scaling the noise variance is as follows: The predicted noise is recalibrated in amplitude to obtain the variance-matched noise. The specific steps include:
[0025] The self-attention matrix is extracted from the multi-layer multi-head self-attention module of the pre-trained diffusion model. First, multi-head averaging is performed within the same layer, and then averaging is performed between layers to obtain the global spatial affinity matrix. ;
[0026] Attention maps are extracted from multi-layer, multi-head cross-attention modules. Channels corresponding to the preset text selection vector are weighted and aggregated, and averaged between layers to obtain an initial spatial relevance vector related to the preset text. ;
[0027] By propagating the correlation vector using the spatial affinity matrix, the structure-enhanced correlation vector is obtained. ;
[0028] For correlation vector Perform min-max normalization to obtain According to the preset mask coverage ratio ,Pick Top- The element corresponding to the position with the largest response is set to 1, and the rest are set to 0, generating a binary space mask. and reshaped into Spatial shape;
[0029] The noise variance scaling factor Expanded to a scaled tensor on the channel dimension, and based on the spatial mask. Local scaling of the predicted noise yields... in This indicates element-wise multiplication, with noise variance correction performed only at positions where the mask is 1.
[0030] As a preferred technical solution, the method of obtaining the time step using variance-matched noise is described. Corresponding inversion latent variables Before the steps, it also includes:
[0031] Using the variance-matched noise For the current latent variables Perform an inversion update to obtain the preheating latent variables.
[0032] by Given the initial state, at a given time step Perform K rounds of noise prediction and inversion refinement to obtain the refined noise. Then use Perform the final inversion update to obtain , where K is the preset number of refinement steps.
[0033] The present invention also provides an image rapid generation apparatus, the apparatus comprising:
[0034] The encoding module is used to obtain latent variables from the input image through a pre-trained variational autoencoder. ;
[0035] Noise prediction module, used to predict based on the latent variables Constructing time steps Corresponding forward reference latent variable and the inversion latent variables corresponding to the previous time step ; and based on the aforementioned forward reference latent variable and the inversion latent variables Obtain the predicted noise;
[0036] The variance matching module performs variance statistics on the predicted noise to obtain a variance scaling factor, and performs amplitude recalibration on the predicted noise based on the variance scaling factor to obtain the variance-matched noise.
[0037] The inversion update module uses the noise after variance matching to obtain the time step. Corresponding inversion latent variables The inversion latent variable trajectory is obtained until time step t > preset inversion time step T. And based on the inversion latent variable trajectory Generate an image.
[0038] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described rapid image generation method.
[0039] The present invention also provides a computer device, the computer device comprising: a processor; and a memory storing a computer program, wherein when the computer program is executed by the processor, the above-described rapid image generation method is implemented.
[0040] This invention proposes a method and apparatus for rapid image generation. Without altering the parameters of the pre-trained diffusion model, it recalibrates the noise amplitude by explicitly aligning the variances of the current predicted noise and the forward reference noise during the inversion process. This fundamentally corrects the problem of underestimation of noise amplitude commonly found in DDIM inversion, effectively suppresses "partial denoising" behavior, and makes the inverted latent variable trajectory more statistically close to the true forward diffusion distribution. Thus, high-fidelity reconstruction is achieved with fewer steps, semantic drift is reduced, and the number of U-Net calls during the inversion process is significantly reduced. Attached Figure Description
[0041] Figure 1 This is a flowchart of a fast image generation method proposed in the first embodiment of the present invention;
[0042] Figure 2 This is a flowchart of a fast image generation method proposed in the second embodiment of the present invention;
[0043] Figure 3This is a structural diagram of an image rapid generation device proposed in the third embodiment of the present invention. Detailed Implementation
[0044] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings. It should be understood that the following embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.
[0045] See Figure 1 The first embodiment of the present invention provides a schematic flowchart of a method for rapid image generation. For example... Figure 1 As shown, the method includes:
[0046] Step 101: Obtain latent variables from the input image using a pre-trained variational autoencoder. ;
[0047] Step 102: Based on the latent variables Constructing time steps Corresponding forward reference latent variable and the inversion latent variables corresponding to the previous time step ; and based on the aforementioned forward reference latent variable and the inversion latent variables Obtain the predicted noise;
[0048] Step 103: Perform variance statistics on the predicted noise to obtain the variance scaling factor, and recalibrate the amplitude of the predicted noise according to the variance scaling factor to obtain the variance-matched noise;
[0049] Step 104: Obtain the time step using the variance-matched noise. Corresponding inversion latent variables The inversion latent variable trajectory is obtained until time step t > preset inversion time step T. And based on the inversion latent variable trajectory Generate an image.
[0050] This method corrects the underestimation of noise amplitude in DDIM inversion by explicitly matching the variance of predicted noise and forward reference noise during the inversion process without changing the parameters of the pre-trained diffusion model. This results in high-fidelity inversion results with fewer steps, reduces semantic drift, and significantly reduces the number of U-Net calls during the inversion process.
[0051] The above embodiments can have several preferred embodiments, as detailed below:
[0052] As a preferred technical solution, the step of basing the latent variable Creating time steps Corresponding forward reference latent variable The steps include:
[0053] At time step Sampling Gaussian noise According to the forward diffusion formula Constructing time steps Corresponding forward reference latent variable ,in and These represent the noise-free component coefficient and the noise component coefficient at time step t, respectively.
[0054] As a preferred technical solution, the step of using the forward reference latent variable... and the inversion latent variables The steps of obtaining predicted noise, performing variance statistics on the predicted noise to obtain a variance scaling factor, and recalibrating the predicted noise based on the variance scaling factor to obtain variance-matched noise include:
[0055] The inversion latent variables are respectively With forward reference latent variables Input a preset noise prediction network The predicted noise is obtained. ;in, For noise prediction based on the forward reference trajectory;
[0056] In the channel dimension, variance statistics are performed on the response values of each channel at all positions in both the batch and spatial dimensions to obtain a prediction noise variance tensor of size 1×C×1×1. and reference noise variance tensor ;
[0057] Calculate the noise variance scaling factor based on the difference between the predicted noise variance and the reference noise variance. , To control the hyperparameters of the correction intensity;
[0058] Based on the noise variance scaling factor The predicted noise is recalibrated in amplitude to obtain the variance-matched noise. .
[0059] As a preferred technical solution, the method of obtaining the time step using variance-matched noise is described. Corresponding inversion latent variables The inversion latent variable trajectory is obtained until time step t > preset inversion time step T. The steps include:
[0060] noise after variance matching According to the DDIM inversion update formula Calculate the time step Inversion latent variables and will Increase by 1, let Until time step t > preset inversion time step T, the final inversion latent variables are obtained. As the endpoint of the inversion, As an inversion latent variable trajectory.
[0061] As a preferred technical solution, the variance matching strength control parameter Dynamically adjusted over time steps; or / and; used to construct the forward distribution of the reference latent variable as follows: The above and These represent the noise-free component coefficient and the noise component coefficient at time step t, respectively.
[0062] As a preferred technical solution, the step of scaling the noise variance is as follows: The predicted noise is recalibrated in amplitude to obtain the variance-matched noise. The specific steps include:
[0063] The self-attention matrix is extracted from the multi-layer multi-head self-attention module of the pre-trained diffusion model. First, multi-head averaging is performed within the same layer, and then averaging is performed between layers to obtain the global spatial affinity matrix. ;
[0064] Attention maps are extracted from multi-layer, multi-head cross-attention modules. Channels corresponding to the preset text selection vector are weighted and aggregated, and averaged between layers to obtain an initial spatial relevance vector related to the preset text. ;
[0065] By propagating the correlation vector using the spatial affinity matrix, the structure-enhanced correlation vector is obtained. ;
[0066] For correlation vector Perform min-max normalization to obtain According to the preset mask coverage ratio ,Pick Top- The element corresponding to the position with the largest response is set to 1, and the rest are set to 0, generating a binary space mask. and reshaped into Spatial shape;
[0067] The noise variance scaling factor Expanded to a scaled tensor on the channel dimension, and based on the spatial mask. Local scaling of the predicted noise yields... in This indicates element-wise multiplication, with noise variance correction performed only at positions where the mask is 1.
[0068] As a preferred technical solution, the method of obtaining the time step using variance-matched noise is described. Corresponding inversion latent variables Before the steps, it also includes:
[0069] Using the variance-matched noise For the current latent variables Perform an inversion update to obtain the preheating latent variables.
[0070] by Given the initial state, at a given time step Perform K rounds of noise prediction and inversion refinement to obtain the refined noise. Then use Perform the final inversion update to obtain Where K is the preset number of refinement steps, wherein... and These represent the noise-free component coefficient and the noise component coefficient at time step t, respectively.
[0071] Through the above preferred embodiments, this invention utilizes one-time noise variance matching to achieve an equivalent replacement for the "landing phase" effect of the ReNoise iterative refinement process. Without relying on multiple iterations, it directly pushes the noise statistics into a near-steady-state range, achieving reconstruction quality close to or better than ReNoise with fewer steps. This significantly reduces the number of U-Net calls during the inversion process and improves computational efficiency. Furthermore, it preserves DDIM... Under the premise of deterministic inversion and skip-step inference characteristics, the original inversion process has been lightly modified. It does not require retraining the diffusion model or introducing additional learnable parameters, and is easy to integrate into the existing diffusion inference framework. It has good engineering implementation and cross-model applicability. An attention-guided spatial masking mechanism is introduced to selectively perform noise variance correction on semantically critical regions and structurally significant regions in space, avoiding excessive perturbation to the background and unimportant regions. This improves the reconstruction quality and editing controllability of key regions while maintaining the consistency of the overall layout and content. The preferred embodiments can be widely applied to application scenarios that rely on high-fidelity inversion, such as image enhancement, image editing, and personalized generation. It can further improve reconstruction accuracy and scalability, and provide a more solid inversion foundation for subsequent high-quality enhancement and controllable editing.
[0072] See Figure 2The second embodiment of the present invention provides a schematic flowchart of a method for rapid image generation. For example... Figure 2 As shown, the method includes:
[0073] Step 201: Diffusion Model Initialization and Latent Variable Acquisition. Specifically, this includes: acquiring the input image to be processed, and inputting the input image into a pre-trained variational autoencoder (VAE) to obtain a noiseless latent variable representation. Load a diffuse noise prediction network that matches the VAE. ,in, For a U-Net structure network pre-trained on a large-scale dataset, its parameters remain constant throughout the entire inversion process of this invention. Simultaneously, a set of diffusion time scheduling parameters consistent with the training process is determined. and inversion steps ,in and These represent the time steps. The corresponding noise-free component coefficients and noise component coefficients.
[0074] Step 202: Construct forward reference samples to obtain reference latent variables at each time step. Specifically, this includes: at each inversion time step... According to the noiseless latent variables Given a time schedule, sample noise from a standard Gaussian distribution.
[0075]
[0076] And utilize forward noise relationship
[0077]
[0078] Constructing time steps Corresponding forward reference latent variable Because this construction is related to the forward diffusion distribution during diffusion model training. Consistent, therefore Statistically consistent with the actual forward diffusion trajectory, it can be used as a reference sample for subsequent noise variance matching.
[0079] Step 203: Perform noise prediction on the reference latent variable and the inverted latent variable, and statistically analyze the difference in noise variance. Specifically, this includes: at the time step... Take the current inversion latent variables , respectively and reference latent variables Input noise prediction network The predicted noise is obtained as follows:
[0080]
[0081] in, For noise prediction based on the inversion trajectory, This is for noise prediction based on the forward reference trajectory. Subsequently, noise prediction is performed on both the channel dimension and the spatial dimension. and Perform variance statistics to obtain the predicted noise variance. variance with reference noise This is used to characterize the difference between the two in terms of noise amplitude.
[0082] Step 204: Calculate the noise variance scaling factor based on the variance difference. Furthermore, local correction can be optionally performed using a spatial mask. Specifically, based on the variance difference obtained in step S203, the noise variance scaling factor is calculated in this embodiment as follows:
[0083]
[0084] in, These are preset variance matching strength control parameters used to balance correction magnitude and numerical stability. Preferably, Can be carried out over time Linear annealing or piecewise settings are used, with smaller values taken at high noise time steps to enhance variance correction and larger values taken at low noise time steps to weaken variance correction.
[0085] In a simple implementation without introducing spatial selection, the predicted noise can be directly recalibrated using the scaling factor to obtain variance-matched noise:
[0086]
[0087] In a preferred embodiment, to avoid excessive perturbation of the background region, the present invention may further introduce an attention-guided spatial mask to spatially constrain the scaling factor: a multi-layer multi-head self-attention matrix is extracted from the self-attention module of the diffusion model and aggregated to obtain a global spatial affinity matrix; then, attention maps related to specific text tokens (e.g., "image") are extracted from the cross-attention module, aggregated to obtain an initial spatial relevance vector, and propagated and normalized using the spatial affinity matrix and Min-Max; subsequently, Top-valued masks are selected according to a preset mask coverage ratio. For each high-response location, set the corresponding spatial position to 1 and the rest to 0 to obtain the binary space mask. Based on this, this embodiment only... A scaling factor is applied to the location to locally recalibrate the predicted noise:
[0088]
[0089] in, This represents element-wise multiplication. In this way, noise variance correction can be applied selectively to semantically critical and structurally significant regions.
[0090] Step 205: Perform DDIM inversion update based on the variance-matched noise, with optional refinement using fewer steps.
[0091] The variance-matched noise obtained in step 204 Substituting the values into the DDIM inversion update formula, a one-step inversion update is performed on the current latent variables to obtain the time step. Inversion latent variables:
[0092]
[0093] In a simplified implementation, the previous update is taken as the time step. The final result. In a more preferred embodiment, the present invention can further refine the inversion result with a few iterations after variance matching is completed: updating the result in one step after variance matching. As the initial state, while maintaining Under the premise of keeping the noise prediction unchanged, perform several iterations of noise prediction and inversion update, and smooth or average the noise prediction results obtained in each iteration with the results of the previous round to obtain the refined noise prediction. Then substitute the values into the DDIM inversion formula to calculate the final result. The number of iterations is much smaller than that used in the ReNoise method, so as to retain most of the reconstruction quality improvement brought about by iterative refinement while significantly reducing computational overhead.
[0094] After completing the update of the current time step, increment the time step index by one, that is, set... and will As input for the next time step, repeat steps S2 to S5 until all inversion time steps are completed.
[0095] Step 206: Output the complete inversion latent variable trajectory for use in low-step denoising reconstruction and downstream editing. Specifically, this includes:
[0096] When time step Exceeding the preset number of inversion steps When the time is up, the inversion process ends, and the final inversion latent variables are output. and the entire trajectory of the inversion latent variables In practical applications, the inverted latent variable trajectory can be used as the starting point for the denoising sampling stage of the diffusion model, reconstructing an image with a high degree of consistency with the structure and semantics of the input image in a low-step DDIM denoising process; furthermore, by introducing text prompts or other prior conditions, the inverted trajectory can be conditionally guided, thereby achieving high-quality image enhancement and controllable image editing while maintaining content consistency.
[0097] This embodiment effectively corrects the "partial denoising" bias caused by the underestimation of noise amplitude in DDIM inversion by explicitly aligning the variance of predicted noise and forward reference noise at each time step without changing the parameters of the pre-trained diffusion model. It achieves high-fidelity inversion with fewer steps and significantly reduces the number of U-Net calls.
[0098] See Figure 3 The third embodiment of the present invention provides a schematic diagram of the structure of an image rapid generation device. (The above...) Figure 1 and Figure 2 The explanations and descriptions of the method embodiments can be applied to this embodiment. For example... Figure 3 As shown, the device includes:
[0099] Encoding module 301 is used to obtain latent variables from the input image through a pre-trained variational autoencoder. ;
[0100] Noise prediction module 302, used to predict based on the latent variables Constructing time steps Corresponding forward reference latent variable and the inversion latent variables corresponding to the previous time step ; and based on the aforementioned forward reference latent variable and the inversion latent variables Obtain the predicted noise;
[0101] The variance matching module 303 performs variance statistics on the predicted noise to obtain a variance scaling factor, and performs amplitude recalibration on the predicted noise based on the variance scaling factor to obtain the variance-matched noise.
[0102] Inversion update module 304 uses the noise after variance matching to obtain the time step. Corresponding inversion latent variables The inversion latent variable trajectory is obtained until time step t > preset inversion time step T. And based on the inversion latent variable trajectory Generate an image.
[0103] This invention proposes a rapid image generation device that, without altering the parameters of the pre-trained diffusion model, recalibrates the noise amplitude by explicitly aligning the variances of the current predicted noise and the forward reference noise during the inversion process. This fundamentally corrects the problem of underestimation of noise amplitude commonly found in DDIM inversion, effectively suppresses "partial denoising" behavior, and makes the inverted latent variable trajectory more statistically close to the true forward diffusion distribution. Thus, high-fidelity reconstruction is achieved with fewer steps, semantic drift is reduced, and the number of U-Net calls during the inversion process is significantly reduced.
[0104] The present invention also provides a computer-readable storage medium (not shown) storing a computer program, which, when executed by a processor, implements the above-described rapid image generation method. This computer-readable storage medium possesses the corresponding technical effects of the above-described rapid image generation method, which will not be elaborated further here.
[0105] The present invention also provides a computer device (not shown), comprising: a processor; and a memory storing a computer program, wherein when the computer program is executed by the processor, the above-described rapid image generation method is implemented. This computer device possesses the corresponding technical effects of the above-described rapid image generation method, which will not be elaborated further here.
[0106] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.
[0107] While this application provides method operation steps as shown in the embodiments or flowcharts, more or fewer operation steps may be included based on conventional or non-inventive labor. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only execution order. In actual device or client product execution, the method can be executed sequentially as shown in the embodiments or drawings, or in parallel (e.g., in a parallel processor or multi-threaded processing environment).
[0108] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0109] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0110] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0111] Specific embodiments have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.
Claims
1. A method for rapid image generation, characterized in that, include: The input image is processed by a pre-trained variational autoencoder to obtain latent variables. ; According to the latent variables Constructing time steps Corresponding forward reference latent variable and the inversion latent variables corresponding to the previous time step ; And based on the aforementioned forward reference latent variable and the inversion latent variables Obtain the predicted noise; The variance of the predicted noise is statistically analyzed to obtain a variance scaling factor, and the predicted noise is recalibrated based on the variance scaling factor to obtain the variance-matched noise. The time step is obtained using the variance-matched noise. Corresponding inversion latent variables The inversion latent variable trajectory is obtained until time step t > preset inversion time step T. And based on the inversion latent variable trajectory Generate an image.
2. The image rapid generation method according to claim 1, characterized in that, According to the latent variable Constructing time steps Corresponding forward reference latent variable The steps include: At time step Sampling Gaussian noise According to the forward diffusion formula , in and These represent the noise-free component coefficient and the noise component coefficient at time step t, respectively. Constructing time steps Corresponding forward reference latent variable .
3. The image rapid generation method according to claim 2, characterized in that, The according to the forward reference latent variable and the inversion latent variables The steps of obtaining predicted noise, performing variance statistics on the predicted noise to obtain a variance scaling factor, and recalibrating the predicted noise based on the variance scaling factor to obtain variance-matched noise include: The inversion latent variables are respectively With forward reference latent variables Input a preset noise prediction network The predicted noise is obtained. ;in, For noise prediction based on the forward reference trajectory; In the channel dimension, variance statistics are performed on the response values of each channel at all positions in both the batch and spatial dimensions to obtain a prediction noise variance tensor of size 1×C×1×1. and reference noise variance tensor ; Calculate the noise variance scaling factor based on the difference between the predicted noise variance and the reference noise variance. , To control the hyperparameters of the correction intensity; , Based on the noise variance scaling factor The predicted noise is recalibrated in amplitude to obtain the variance-matched noise. .
4. The image rapid generation method according to claim 3, characterized in that, The time step is obtained by using the noise after variance matching. Corresponding inversion latent variables The inversion latent variable trajectory is obtained until time step t > preset inversion time step T. The steps include: noise after variance matching According to the DDIM inversion update formula: , Calculate the time step Inversion latent variables and will Increase by 1, let Until time step t > preset inversion time step T, the final inversion latent variables are obtained. As the endpoint of the inversion, As an inversion latent variable trajectory.
5. The image rapid generation method according to claim 3 or 4, characterized in that, The hyperparameters With time step Dynamic adjustment is performed using a preset annealing strategy; or / and; The forward distribution used to construct the forward reference latent variable .
6. The image rapid generation method according to claim 3 or 4, characterized in that, The scaling factor based on the noise variance The predicted noise is recalibrated in amplitude to obtain the variance-matched noise. The specific steps include: The self-attention matrix is extracted from the multi-layer multi-head self-attention module of the pre-trained diffusion model. First, multi-head averaging is performed within the same layer, and then averaging is performed between layers to obtain the global spatial affinity matrix. ; Attention maps are extracted from multi-layer, multi-head cross-attention modules. Channels corresponding to the preset text selection vector are weighted and aggregated, and averaged between layers to obtain an initial spatial relevance vector related to the preset text. ; By propagating the correlation vector using the spatial affinity matrix, the structure-enhanced correlation vector is obtained. ; For correlation vector Perform min-max normalization to obtain According to the preset mask coverage ratio ,Pick Top- The element corresponding to the position with the largest response is set to 1, and the rest are set to 0, generating a binary space mask. and reshaped into Spatial shape; The noise variance scaling factor Expanded to a scaled tensor on the channel dimension, and based on the spatial mask. By locally scaling the predicted noise, we obtain: , in This indicates element-wise multiplication, with noise variance correction performed only at positions where the mask is 1.
7. The image rapid generation method according to any one of claims 1 to 4, characterized in that, The time step is obtained by using the noise after variance matching. Corresponding inversion latent variables Before the steps, it also includes: Using the variance-matched noise For the current latent variables Perform an inversion update to obtain the preheating latent variables: , by Given the initial state, at a given time step Perform K rounds of noise prediction and inversion refinement to obtain the refined noise. Then use Perform the final inversion update to obtain , where K is the preset number of refinement steps.
8. An image rapid generation apparatus, characterized in that, The device includes: The encoding module is used to obtain latent variables from the input image through a pre-trained variational autoencoder. ; Noise prediction module, used to predict based on the latent variables Constructing time steps Corresponding forward reference latent variable and the inversion latent variables corresponding to the previous time step ; and based on the aforementioned forward reference latent variable and the inversion latent variables Obtain the predicted noise; The variance matching module performs variance statistics on the predicted noise to obtain a variance scaling factor, and performs amplitude recalibration on the predicted noise based on the variance scaling factor to obtain the variance-matched noise. The inversion update module uses the noise after variance matching to obtain the time step. Corresponding inversion latent variables The inversion latent variable trajectory is obtained until time step t > preset inversion time step T. And based on the inversion latent variable trajectory Generate an image.
9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the image rapid generation method as described in any one of claims 1 to 7.
10. A computer device, characterized in that, The computer device includes: processor; and A memory storing a computer program that, when executed by a processor, implements the rapid image generation method as described in any one of claims 1 to 7.