Fine-tuning-free diffusion model acceleration method and system based on residual cache

Abstract

The invention discloses a fine-tuning-free diffusion model acceleration method and system based on residual cache, and the method comprises the steps: dividing the calculation time step of a diffusion model into a dense calculation stage and a sparse calculation stage for the diffusion model in diffusion reverse denoising; in the dense calculation stage, complete forward calculation is carried out on all primitives of input images or video data, features of the primitives are cached, and meanwhile the primitives are divided into a plurality of clusters through a trajectory clustering method based on time sequence enhancement; in the sparse calculation stage, one proxy primitive is selected from each cluster for real denoising, and updating of other proxy primitives in the same cluster is deduced by utilizing a denoising simulation method based on proxy guidance, so that diffusion model acceleration is realized. According to the system, the calculation amount and the reasoning time of diffusion model reasoning can be remarkably reduced on the premise that the generation quality and the semantic alignment performance are not lost basically.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence and diffusion model acceleration, specifically to a method and system for accelerating diffusion models without fine-tuning based on residual caching. Background Technology

[0002] Diffusion models have become a crucial foundational technology in generative artificial intelligence due to their superior performance in tasks such as high-fidelity image synthesis, video generation, image editing, super-resolution, image inpainting, and multimodal conditional generation. In recent years, diffusion models have rapidly developed in high-fidelity image and video generation tasks, continuously improving generation quality and semantic alignment capabilities by increasing model capacity, input resolution, and sequence length. However, this trade-off of larger models and longer sequences for higher quality significantly increases the computational cost per sampling, resulting in slower inference speeds and higher memory consumption. This puts considerable pressure on interactive generation, online services, and deployment on resource-constrained devices. Therefore, efficient inference acceleration methods for diffusion models are needed.

[0003] In the area of ​​accelerating diffusion models, existing research can be broadly categorized into three types. Sampler-based methods reduce denoising steps and compress the sampling process by improving time step scheduling, constructing higher-order numerical solvers, distillation, and consistency modeling. Model structure-based methods reduce operator overhead and parameter size in each denoising step through pruning, quantization, or reconstructing the network architecture, as well as feature-based acceleration methods.

[0004] Feature-based acceleration methods directly target intermediate features or primitives, including primitive pruning, primitive merging, and feature caching strategies. Primitive merging methods approximate the original representation by merging similar primitives and using hybrid features, reducing the number of primitives involved in attention and feedforward computations. Feature caching methods reuse some primitive features between adjacent time steps, avoiding redundant computations on primitives that change little. These methods effectively reduce computational redundancy in spatial or temporal dimensions, but most only measure feature similarity at a single time step or directly transfer historical features to subsequent time steps, lacking explicit modeling of denoising trajectories and residual evolution patterns. At higher speedup ratios, they are prone to problems such as accumulated errors, detail degradation, or decreased stability.

[0005] Therefore, there is an urgent need to study a diffusion model acceleration method based on residual caching, and to redesign the feature-level acceleration process from the perspective of "residual reusability". Summary of the Invention

[0006] To address the problems existing in the prior art, this application proposes a diffusion model acceleration method based on residual caching to accelerate the inference process of the diffusion model without retraining the model.

[0007] In a first aspect, embodiments of this application provide a method for accelerating a fine-tuning-free diffusion model based on residual caching, applicable to diffusion models for image and video generation. The method includes:

[0008] Time step division steps: For diffusion model in diffusion reverse denoising, the time step of diffusion model calculation is divided into dense calculation stage and sparse calculation stage.

[0009] The trajectory clustering steps based on temporal enhancement are as follows: In the intensive computation stage, all primitives of the input image or video data are fully computed forward and their feature trajectories are cached. At the same time, the primitives are divided into several clusters by the trajectory clustering method based on temporal enhancement.

[0010] The denoising simulation steps based on surrogate guidance are as follows: In the sparse computation stage, a surrogate primitive is selected from each cluster for real denoising, and the update of the other surrogate primitives in the same cluster is deduced using the surrogate-guided denoising simulation method, thereby accelerating the diffusion model.

[0011] In this embodiment of the invention, the above-mentioned trajectory clustering step based on temporal enhancement includes:

[0012] Step-by-step similarity calculation: Input image or video data with a multidimensional feature trajectory sequence of all primitives at each time step, calculate the cosine similarity between each pair of primitives, and construct the corresponding similarity matrix;

[0013] The similarity accumulation step based on time-series enhancement is as follows: a moving average mechanism in the time direction is introduced into the time-step similarity matrix to obtain the cumulative similarity matrix;

[0014] The steps of trajectory clustering based on similarity are as follows: The primitives are grouped and clustered using a cosine distance-based clustering method to generate a set of clusters. Based on the cumulative similarity matrix, the clustering results are optimized using an objective function. For each cluster, a primitive is randomly selected as a surrogate primitive. The remaining primitives in the cluster, excluding the surrogate primitive, are collectively referred to as the set of surrogate primitives.

[0015] In this embodiment of the invention, the above-mentioned agent-guided denoising simulation step includes:

[0016] Denoising proxy primitive steps: Through the denoising diffusion model, the proxy primitive at the time step is updated by the features of the next time step. Based on the features of the primitive in the previous time step, a discrete feature trajectory is obtained as the proxy trajectory, and a multi-order residual is constructed based on the recursive finite difference.

[0017] The steps for estimating the residuals of the proxied primitives are as follows: Based on the multi-order residuals of the proxy trajectory, a confidence weight related to the order is constructed to measure the alignment between the proxy trajectory and the proxied trajectory in the multi-order residuals. Then, the multi-order residuals of the proxied primitives are estimated at the previous time step. Then, the multi-order residuals of the proxied primitives at the previous time step and the multi-order residuals of the proxy trajectory at the current time step are weighted and fused to estimate the multi-order residuals of the proxied primitives at the current time step.

[0018] The denoising step of the proxied primitive simulation is as follows: For the multi-order residuals, the state of the proxied primitives is updated using implicit ODE numerical solution methods of different orders or forms, so as to realize the denoising of the proxied primitive simulation.

[0019] In this embodiment of the invention, the moving average mechanism in the above-mentioned time-series-enhanced similarity accumulation step includes: calculating the cumulative similarity matrix. :

[0020]

[0021] in, The similarity matrix at each time step [0,1] is used to control the weight of the current time step and its subsequent time steps in the cumulative similarity; The closer it gets to the upper limit, the higher the similarity ratio of the current time step, thus strengthening the focus on the current residual pattern; The closer it gets to the lower limit, the more the cumulative similarity focuses on historical trajectories.

[0022] In this embodiment of the invention, the objective function in the above-mentioned trajectory clustering step based on similarity is:

[0023]

[0024] Where N is the number of primitives and K is the number of clusters. For time steps For indicator functions, For primitives, when Image element Classified into clusters The time value is Otherwise ; Represents primitives under the cumulative similarity metric. with cluster Cosine similarity between centers.

[0025] In this embodiment of the invention, in the above-described step of estimating the residuals of the proxied primitives, the confidence weight... for:

[0026]

[0027] in, In time step The estimated proximate primitives are the first order residual , In time step The first estimated proxy primitive order residual , The closer the value is to This indicates that the two trajectories are in the first... The more consistent the directions on the residuals, the better.

[0028] In this embodiment of the invention, in the above-described step of estimating the residuals of the proxied primitives, at the time step... The estimated proximate primitives are the first order residual The weighted fusion of its own residual and the proxy residual is expressed as:

[0029]

[0030] in, For the proxied primitive at time step The basic residual direction, For the agent trajectory at time step The introduced time-look-ahead correction term, , where is the confidence weight.

[0031] Secondly, embodiments of this application provide a fine-tuning-free diffusion model acceleration system based on residual caching. The system employs the aforementioned fine-tuning-free diffusion model acceleration method based on residual caching, applied to diffusion models for image and video generation. The system includes:

[0032] Time step partitioning module: used to divide the time step of diffusion model computation into dense computation stage and sparse computation stage in diffusion inverse denoising.

[0033] The temporally enhanced trajectory clustering module is used to perform complete forward computation on all primitives of the input image or video data and cache their feature trajectories during the intensive computation phase. At the same time, it divides the primitives into several clusters through a temporally enhanced trajectory clustering method.

[0034] The proxy-guided denoising simulation module is used to select a proxy primitive from each cluster for real denoising during the sparse computation stage, and to use the proxy-guided denoising simulation method to deduce the updates of the other proxied primitives in the same cluster, thereby accelerating the diffusion model.

[0035] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of a residual cache-based fine-tuning diffusion model acceleration method.

[0036] Fourthly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of the fine-tuning diffusion model acceleration method based on residual cache as described above.

[0037] Compared with existing technologies, it has the following outstanding advantages:

[0038] 1) The method and system of this invention propose a residual caching strategy based on intensive and sparse computation stages: In the diffusion reverse process, this invention divides the time step into an intensive computation stage and a sparse computation stage: the former performs complete forward computation on all primitives and caches intermediate features, while the latter only performs real denoising on proxy primitives in each cluster and uses simulation to update primitives in the same cluster. This strategy reduces redundant computation in later time steps and improves inference efficiency without changing the original model structure and parameters.

[0039] 2) The method and system of this invention propose a trajectory clustering method based on temporal enhancement: Based on historical features of multiple time steps, this invention first calculates the similarity step by step, then obtains a temporally enhanced similarity matrix through time-weighted accumulation, and uses clustering with cosine distance as the metric to divide primitives with similar residual evolution patterns into clusters and select surrogate primitives. This method can more stably identify primitive groups with similar denoised trajectories, improving the reliability of subsequent surrogate simulations.

[0040] 3) The method and system of this invention propose a denoising simulation method based on surrogate guidance: In the sparse computation stage, this invention constructs a multi-order residual representation for surrogate primitives, calculates confidence weights based on the residual similarity between the surrogate trajectory and the trajectory of the proxied primitive, and then performs weighted fusion of the two residuals before substituting them into the implicit ODE numerical format to simulate denoising of the proxied primitives. This method, with less real forward computation, can still maintain an update direction and numerical stability close to the original diffusion process.

[0041] 4) The method and system of this invention propose a training-free deployment method compatible with diffusion models and numerical solvers: The residual caching framework of this invention is embedded into the existing diffusion model inference process as an external module, only adding operations such as caching, clustering, and implicit ODE updates, without involving model weight modification or retraining. By adjusting the stage division, the number of clusters, and the ODE order, this method can adapt to diffusion models with different structures and scheduling strategies, facilitating rapid deployment on various tasks and datasets. Attached Figure Description

[0042] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0043] Figure 1 This is a schematic diagram of the fine-tuning-free diffusion model acceleration method based on residual caching according to the present invention;

[0044] Figure 2 This is a schematic diagram of the fine-tuning-free diffusion model acceleration method based on residual caching in an embodiment of the present invention;

[0045] Figure 3 This is a schematic diagram of the fine-tuning-free diffusion model acceleration system based on residual caching according to the present invention;

[0046] Figure 4 This is a schematic diagram of the computer hardware of the present invention. Detailed Implementation

[0047] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0048] It should also be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0049] It should also be understood that, in various embodiments of the present invention, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0050] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0051] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0052] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0053] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0054] To make the above-mentioned features and effects of the present invention clearer and easier to understand, specific embodiments are described below in conjunction with the accompanying drawings. This specification discloses one or more embodiments incorporating the features of the present invention. The disclosed embodiments are merely illustrative. The scope of protection of the present invention is not limited to the disclosed embodiments, but is defined by the appended claims.

[0055] The following are system embodiments corresponding to the above method embodiments. This embodiment can be implemented in conjunction with the above embodiments. The relevant technical details mentioned in the above embodiments are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiments.

[0056] This invention takes into account that primitives with similar denoised trajectories often maintain a high degree of consistency in the direction and magnitude of feature changes (residuals) between adjacent time steps, especially the lower-order residuals which have stronger reusability; at the same time, the reliability of such reusable residuals can be estimated by the relationship between the trajectories of the previous several time steps.

[0057] Based on this discovery, this invention proposes a diffusion model acceleration method based on residual caching. It redesigns the feature-level acceleration process from the perspective of "residual reusability": during the intensive computation phase, it caches the historical trajectories of primitives across multiple time steps and constructs primitive clusters with similar residual evolution patterns using a time-enhanced trajectory clustering method; during the sparse computation phase, it performs true denoising only on surrogate primitives within each cluster and constructs a surrogate-guided denoising simulation method based on their multi-order residuals, simulating denoising updates for the remaining primitives in the same cluster within an implicit ordinary differential equation framework. This design preserves the feature representation of each primitive while utilizing surrogate residuals to provide "updated" correction directions, achieving efficient and stable inference acceleration for diffusion models without additional training, and is compatible with various existing diffusion models and acceleration frameworks.

[0058] The system of this application embodiment will be described in detail below with reference to specific embodiments:

[0059] like Figure 1 and Figure 2 As shown, this invention proposes a diffusion model acceleration method based on residual caching. In the diffusion reverse denoising process, this method divides the time step into a dense computation stage and a sparse computation stage. The dense computation stage refers to the stage where all primitives need to be computed, while the sparse computation stage refers to the stage where only some primitives need to be computed. In the dense computation stage, all primitives undergo complete forward computation and their feature trajectories are cached. Simultaneously, a trajectory clustering method based on temporal enhancement is used to divide primitives with similar residual evolution patterns into several clusters. In the sparse computation stage, only one surrogate primitive from each cluster is selected for actual denoising. Then, a surrogate-guided denoising simulation method is used to explicitly deduce the updates of the remaining surrogate primitives in the same cluster within an implicit ordinary differential equation framework, thereby significantly reducing redundant computation while maintaining generation quality.

[0060] The method includes the following steps:

[0061] Step 101: For the diffusion model in diffusion reverse denoising, the time step of the diffusion model calculation is divided into a dense calculation stage and a sparse calculation stage.

[0062] Step 102 of the trajectory clustering based on temporal enhancement: In the intensive computation stage, all primitives of the input image or video data are fully computed forward and their feature trajectories are cached. At the same time, the primitives are divided into several clusters by the trajectory clustering method based on temporal enhancement.

[0063] Step 103 of the Proxy-Guided Denoising Simulation: In the sparse computation stage, a proxy primitive is selected from each cluster for real denoising, and the Proxy-Guided Denoising Simulation Method is used to deduce the update of the other proxied primitives in the same cluster, thereby accelerating the diffusion model.

[0064] In this embodiment of the invention, the above-mentioned trajectory clustering step 102 based on temporal enhancement includes:

[0065] Step-by-step similarity calculation: Input image or video data with a multidimensional feature trajectory sequence of all primitives at each time step, calculate the cosine similarity between each pair of primitives, and construct the corresponding similarity matrix;

[0066] The similarity accumulation step based on time-series enhancement is as follows: a moving average mechanism in the time direction is introduced into the time-step similarity matrix to obtain the cumulative similarity matrix;

[0067] The steps of trajectory clustering based on similarity are as follows: The primitives are grouped and clustered using a cosine distance-based clustering method to generate a set of clusters. Based on the cumulative similarity matrix, the clustering results are optimized using an objective function. For each cluster, a primitive is randomly selected as a surrogate primitive. The remaining primitives in the cluster, excluding the surrogate primitive, are collectively referred to as the set of surrogate primitives.

[0068] In this embodiment of the invention, the moving average mechanism in the above-mentioned time-series-enhanced similarity accumulation step includes: calculating the cumulative similarity matrix. :

[0069]

[0070] in, The similarity matrix at each time step This is a smoothing factor used to control the weight of the current time step and its subsequent time steps in the cumulative similarity; when When the similarity is greater than the preset value, the similarity ratio at the current time step is higher, strengthening the focus on the current residual pattern; when When the cumulative similarity is less than the preset value, the longer historical trajectories are smoothly integrated.

[0071] In this embodiment of the invention, the objective function in the above-mentioned trajectory clustering step based on similarity is:

[0072]

[0073] Where N is the number of primitives and K is the number of clusters. For time steps For indicator functions, For primitives, when Image element Classified into clusters The time value is Otherwise ; Represents primitives under the cumulative similarity metric. with cluster Cosine similarity between centers.

[0074] Specifically, such as Figure 2 As shown in the specific embodiment of the present invention, the above-mentioned trajectory clustering step 102 based on temporal enhancement specifically includes:

[0075] like Figure 2 As shown in the blue module in the lower left corner, this embodiment clusters primitives based on historical denoised trajectories so that a small number of surrogate primitives can represent a cluster of primitives with similar residual evolution behavior in subsequent time steps. Considering the continuous evolution of primitive features over time in the diffusion model and the stronger indicative role of the most recent time step for future residuals, this embodiment proposes a Temporally Enhanced Trajectory Clustering (TETC) method. This method includes three steps: ① calculating time-step similarity, ② similarity accumulation based on temporal enhancement, and ③ trajectory clustering based on similarity.

[0076] ① Calculate time-step similarity: Given a sequence of feature trajectories of a network layer over a historical time range. ,in Indicates at time step All Each primitive 3D feature representation, The number of primitives, The feature dimension is used. In this embodiment, the cosine similarity between any two primitives is first calculated at each time step, and the corresponding similarity matrix is ​​constructed. :

[0077] ,

[0078] Among them, matrix The The item indicates the time step. Upper The primitive and the first The cosine similarity between primitives is used to characterize the degree of closeness between them in the residual information at the current time step.

[0079] ② Time-series-enhanced similarity accumulation: To enhance the influence of recent time steps on clustering results and smooth noise in historical trajectories, this embodiment adds a similarity accumulation matrix to the time-step similarity matrix. Based on this, a moving average mechanism in the time direction is introduced to obtain the cumulative similarity matrix. :

[0080]

[0081] in, This is a smoothing factor used to control the weight of the current time step and its subsequent time steps in the cumulative similarity. When the similarity is larger, the proportion of similarity at the current time step is higher, thus strengthening the focus on the most recent residual pattern; when When the value is small, the accumulated similarity can smoothly integrate longer historical trajectories, thereby improving the stability of the clustering results.

[0082] ③ Similarity-based trajectory clustering: Intermediate features in diffusion models are often high-dimensional and relatively sparse. In such scenarios, distance metrics based on cosine similarity can better reflect the similarity relationships along feature directions. Therefore, this embodiment uses the K-medoids clustering method based on cosine distance to group primitives. Specifically, given all... Let there be Graph elements, and denote the set of clusters as . In this embodiment, the clustering results are optimized using the following objective function:

[0083]

[0084] in, As an indicator function, when the primitive Classified into clusters The time value is Otherwise ; Represents primitives under the cumulative similarity metric. with cluster Cosine similarity between centers. The K-medoids algorithm constrains the center of each cluster to be an actual sample point, thus ensuring the physical interpretability of subsequent proxy primitives. For each cluster... In this embodiment, a primitive is randomly selected from the cluster as a proxy primitive, denoted as . ,satisfy All primitives within a cluster, excluding the proxy primitives, are collectively referred to as the set of proxied primitives. Therefore, ,and In the subsequent sparse computation stage, only the proxy primitives undergo actual network forward propagation and denoising computation, while the updates of the proxied primitives are completed by the proxy-guided denoising simulation module.

[0085] In this embodiment of the invention, the above-mentioned agent-guided denoising simulation step 103 includes:

[0086] Denoising proxy primitive steps: Through the denoising diffusion model, the proxy primitive at the time step is updated by the features of the next time step. Based on the features of the primitive in the previous time step, a discrete feature trajectory is obtained as the proxy trajectory, and a multi-order residual is constructed based on the recursive finite difference.

[0087] The steps for estimating the residuals of the proxied primitives are as follows: Based on the multi-order residuals of the proxy trajectory, a confidence weight related to the order is constructed to measure the alignment between the proxy trajectory and the proxied trajectory in the multi-order residuals. Then, the multi-order residuals of the proxied primitives are estimated at the previous time step. Then, the multi-order residuals of the proxied primitives at the previous time step and the multi-order residuals of the proxy trajectory at the current time step are weighted and fused to estimate the multi-order residuals of the proxied primitives at the current time step.

[0088] The denoising step of the proxied primitive simulation is as follows: For the multi-order residuals, the state of the proxied primitives is updated using implicit ODE numerical solution methods of different orders or forms, so as to realize the denoising of the proxied primitive simulation.

[0089] In this embodiment of the invention, in the above-described step of estimating the residuals of the proxied primitives, the confidence weight... for:

[0090]

[0091] in, In time step The estimated proximate primitives are the first order residual , In time step The first estimated proxy primitive order residual , The closer the value is to This indicates that the two trajectories are in the first... The more consistent the directions on the residuals, the better.

[0092] In this embodiment of the invention, in the above-described step of estimating the residuals of the proxied primitives, at the time step... The estimated proximate primitives are the first order residual The weighted fusion of its own residual and the proxy residual is expressed as:

[0093]

[0094] in, For the proxied primitive at time step The basic residual direction, For the agent trajectory at time step The introduced time-look-ahead correction term, , where is the confidence weight.

[0095] Specifically, such as Figure 2 As shown in the specific embodiment of the present invention, the above-mentioned agent-guided denoising simulation step 103 specifically includes:

[0096] like Figure 2 As shown in the yellow module in the lower right corner, this embodiment, based on time-enhanced trajectory clustering, uses the denoised trajector ...

[0097] ① Denoising proxy primitives: For each cluster and every layer of the network Record time steps The characteristics of the upper proxy primitive are In the reverse denoising process of the diffusion model, the denoising operator of the network at this layer is denoted as... Then time step The proxy primitive on the time step The features are obtained through a single reverse update:

[0098] ,

[0099] For ease of explanation, the cluster index will be omitted below. With layer index memorize directly Indicates the proxy primitive at time step The characteristics of this are as follows: By updating the proxy primitives multiple times along the time step sequence, a discrete time trajectory can be obtained. To uniformly characterize this trajectory within the framework of ordinary differential equations (ODE), this embodiment constructs multi-order residuals based on recursive finite differences. First, the zeroth-order term is considered as the feature itself:

[0100]

[0101] Then, higher-order residuals are defined using a recursive difference method:

[0102]

[0103] The resulting set It can be viewed as a multi-order "derivative" residual description of the surrogate trajectory at each time step, which will be used in subsequent steps to estimate the residuals of the surrogate primitives in the same cluster, where M represents the order of the multi-order residuals.

[0104] ② Estimate the residuals of the proxied primitives: for each cluster It contains a proxy primitive. and a set of proxied primitives Describe the time step of a proxied primitive. The characteristics are To simplify notation, cluster index and primitive index will be omitted below, and will be denoted as... This indicates that any proxied primitive at time step The characteristics of the surrogate trajectory. The multi-order residuals of the surrogate trajectory have already been obtained in step ①. This embodiment first constructs an order-related confidence weight based on these residuals, which is used to measure the relationship between the surrogate trajectory and the proxied trajectory at the [missing information - likely a specific timeframe]. Alignment degree on the order residuals. The specific definition is as follows:

[0105]

[0106] in The closer the value is to This indicates that the two trajectories are in the first... The more consistent the direction of the residuals, the better. Subsequently, in this embodiment, at time step... The estimated proximate primitives are the first order residual Its form is a weighted fusion of its own residual and the proxy residual:

[0107]

[0108] in, It can be regarded as the proxied primitive at time step The basic residual direction, and This provides the agent trajectory at time step The introduced time-lookahead correction term. Confidence level. The strength of the correction term is determined when the two trajectories are at the same height. near When the surrogate primitives are aligned, the residuals of the surrogate primitives will be strongly aligned to the surrogate residuals; otherwise, they will retain more of their own residual orientation.

[0109] ③ Noise reduction based on implicit ODE for proxied primitives: From the perspective of ordinary differential equations, the trajectory of the proxied primitives... It evolves according to a certain implicit dynamic on the reverse time axis. Unlike explicitly constructing a continuous drift function, this embodiment directly uses the multi-order residuals estimated above. Considered as at time step Discrete approximations of the time derivatives of each order are used, and the state update of the proxied primitives is advanced using a unit step form of the implicit Taylor method. Specifically, the time step... Towards The update formula is:

[0110] .

[0111] when When this happens, the formula degenerates into an implicit Euler form update:

[0112] .

[0113] Furthermore, the estimated multi-order residuals It is also compatible with standard implicit linear multistep ordinary differential equation solution schemes. For example, when using the unit step BDF2 scheme, the update of the surrogate primitive can be written as:

[0114] .

[0115] in For time steps The characteristics of the proxied primitives are shown above. Therefore, the multi-order residuals constructed in this embodiment can not only be used in the implicit Taylor method, but can also be directly embedded into high-order or multi-step implicit ODE solvers without requiring additional modifications to the original diffusion model structure. Those skilled in the art can select different orders or forms of implicit ODE numerical schemes to efficiently and stably simulate and denoise the temporal evolution of the proxied primitives without departing from the core ideas of this invention.

[0116] As described above, the system and method of the present invention can be implemented well.

[0117] Compared with the prior art, the present invention has the following outstanding advantages and beneficial effects:

[0118] This invention offers the following beneficial technical effects: Key Point 1: By using time-stage partitioning and residual caching, the complete forward computation of all primitives is avoided throughout the diffusion process, providing a basic framework for overall acceleration. Key Point 2: Trajectory clustering based on temporal enhancement stably constructs primitive clusters, enabling surrogate primitives to represent the residual evolution patterns of primitives within the same cluster, mitigating the impact of grouping errors on generation quality. Key Point 3: Through surrogate-guided denoising simulation, leveraging multi-order residuals and implicit ODE updates, good image quality and semantic consistency are maintained at a high speedup ratio. Key Point 4: This invention ensures universal, training-free deployment capability, allowing direct application to various existing diffusion models and their numerical solvers. In summary, by combining these four key points, this invention significantly reduces the computational load and inference time of diffusion model inference without substantially sacrificing generation quality and semantic alignment performance, providing an effective acceleration solution for the deployment and application of diffusion models in resource-constrained scenarios.

[0119] Secondly, such as Figure 3 As shown, this application provides a residual caching-based fine-tuning diffusion model acceleration system. It employs the aforementioned residual caching-based fine-tuning diffusion model acceleration method and is applied to diffusion models for image and video generation. The system includes:

[0120] Time step partitioning module 201: used to divide the time step of the diffusion model calculation into a dense calculation stage and a sparse calculation stage in diffusion inverse denoising for the diffusion model;

[0121] The trajectory clustering module 202 based on temporal enhancement is used to perform complete forward computation on all primitives of the input image or video data and cache their feature trajectories during the intensive computation stage. At the same time, it divides the primitives into several clusters through the trajectory clustering method based on temporal enhancement.

[0122] The proxy-guided denoising simulation module 203 is used to select a proxy primitive from each cluster for real denoising during the sparse computation stage, and to use the proxy-guided denoising simulation method to deduce the update of the other proxied primitives in the same cluster, thereby accelerating the diffusion model.

[0123] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the described method for accelerating a non-fine-tuning diffusion model based on residual caching.

[0124] Fourthly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of the fine-tuning diffusion model acceleration method based on residual caching as described above.

[0125] In addition, combined Figure 1 The residual caching-based no-fine-tuning diffusion model acceleration method described in this application embodiment can be implemented by electronic devices, such as computer devices. Figure 4 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of this application.

[0126] In some embodiments, the computer device may further include a communication interface 83 and a bus 80. For example, Figure 4 As shown, the processor 81, memory 82, and communication interface 83 are connected through bus 80 and complete communication with each other.

[0127] Specifically, the processor 81 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0128] The memory 82 can be used to store or cache various data files that need to be processed and / or communicated, as well as possible computer program instructions executed by the processor 81.

[0129] The processor 81 reads and executes computer program instructions stored in the memory 82 to implement any of the fine-tuning diffusion model acceleration methods based on residual cache in the above embodiments.

[0130] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0131] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for accelerating a fine-tuning-free diffusion model based on residual caching, characterized in that, A diffusion model applied to image and video generation, the method comprising: Time step division steps: For the diffusion model in diffusion reverse denoising, the time step of the diffusion model calculation is divided into a dense calculation stage and a sparse calculation stage; The trajectory clustering step based on temporal enhancement is as follows: In the intensive computation stage, all primitives of the input image or video data are fully computed forward and their features are cached. At the same time, the primitives are divided into several clusters by the trajectory clustering method based on temporal enhancement. The proxy-guided denoising simulation steps are as follows: In the sparse computation stage, a proxy primitive is selected from each cluster for real denoising, and the proxy-guided denoising simulation method is used to deduce the updates of the other proxied primitives in the same cluster, thereby accelerating the diffusion model.

2. The method for accelerating the diffusion model without fine-tuning based on residual caching according to claim 1, characterized in that, The time-enhanced trajectory clustering step includes: Step-by-step similarity calculation: Input image or video data and calculate the cosine similarity between each pair of primitives at each time step, and construct the corresponding similarity matrix. The similarity accumulation step based on time-series enhancement is as follows: a moving average mechanism in the time direction is introduced into the similarity matrix step by step to obtain the cumulative similarity matrix; The trajectory clustering steps based on similarity are as follows: a cosine distance-based clustering method is used to group and cluster the primitives to generate a set of clusters. Based on the cumulative similarity matrix, the clustering results are optimized by an objective function. For each cluster, a primitive is randomly selected as a surrogate primitive. The remaining primitives in the cluster, excluding the surrogate primitive, are collectively referred to as the set of surrogate primitives.

3. The method for accelerating the diffusion model without fine-tuning based on residual caching according to claim 1, characterized in that, The agent-guided denoising simulation step includes: Denoising proxy primitive steps: Through the denoising diffusion model, the proxy primitive at the time step is updated by the features of the next time step. Based on the features of the primitive in the previous time step, a discrete feature trajectory is obtained as the proxy trajectory, and a multi-order residual is constructed based on the recursive finite difference. The steps for estimating the residuals of the proxied primitives are as follows: Based on the multi-order residuals of the proxy trajectory, a confidence weight related to the order is constructed to measure the alignment between the proxy trajectory and the proxied trajectory in the multi-order residuals. Then, the multi-order residuals of the proxied primitives are estimated at the previous time step. The multi-order residuals of the proxied primitives at the previous time step are weighted and fused with the multi-order residuals of the proxy trajectory at the current time step to estimate the multi-order residuals of the proxied primitives at the current time step. The denoising step of the proxied primitive simulation is as follows: For the multi-order residual, the state of the proxied primitive is updated by using implicit ODE numerical solution methods of different orders or different forms, so as to realize the denoising of the proxied primitive simulation.

4. The method for accelerating the diffusion model without fine-tuning based on residual caching according to claim 2, characterized in that, The moving average mechanism in the time-enhanced similarity accumulation step includes: Calculate the cumulative similarity matrix : in, The similarity matrix at each time step As a smoothing factor, [0,1] is used to control the weight of the current time step and its subsequent time steps in the cumulative similarity; The closer it gets to the upper limit, the higher the similarity ratio of the current time step, thus strengthening the focus on the current residual pattern; The closer it gets to the lower limit, the more the cumulative similarity focuses on historical trajectories.

5. The method for accelerating the diffusion model without fine-tuning based on residual caching according to claim 2, characterized in that, In the similarity-based trajectory clustering step, the objective function is: Where N is the number of primitives and K is the number of clusters. For time step For indicator functions, For primitives, when Image element Classified into clusters The time value is Otherwise ; Represents primitives under the cumulative similarity metric. with cluster Cosine similarity between centers.

6. The method for accelerating the diffusion model without fine-tuning based on residual caching according to claim 3, characterized in that, In the step of estimating the residuals of the proxied primitives, the confidence weights for: in, In time step The estimated proximate primitives are the first order residual , In time step The first estimated proxy primitive order residual , The closer the value is to This indicates that the two trajectories are in the first... The more consistent the directions on the residuals, the better.

7. The method for accelerating the diffusion model without fine-tuning based on residual caching according to claim 3, characterized in that, In the step of estimating the residuals of the proxied primitives, at the time step The estimated proximate primitives are the first order residual The weighted fusion of its own residual and the proxy residual is expressed as: in, For the proxied primitive at time step The basic residual direction, For the agent trajectory at time step The introduced time-look-ahead correction term, , where is the confidence weight.

8. A residual cache-based no-fine-tuning diffusion model acceleration system, employing the residual cache-based no-fine-tuning diffusion model acceleration method as described in any one of claims 1-7, characterized in that, The system, which is applied to a diffusion model for image and video generation, includes: Time step partitioning module: used to divide the time step of the diffusion model calculation into a dense calculation stage and a sparse calculation stage in diffusion reverse denoising; The trajectory clustering module based on temporal enhancement is used in the intensive computing stage to perform complete forward computation on all primitives of the input image or video data and cache their feature trajectories. At the same time, the primitives are divided into several clusters by the trajectory clustering method based on temporal enhancement. The proxy-guided denoising simulation module is used to select a proxy primitive from each cluster for real denoising during the sparse computation stage, and to use the proxy-guided denoising simulation method to deduce the updates of the remaining proxied primitives in the same cluster, thereby accelerating the diffusion model.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the fine-tuning diffusion model acceleration method based on residual caching as described in any one of claims 1-7.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the fine-tuning diffusion model acceleration method based on residual caching as described in any one of claims 1-7.

Images