A trajectory-decomposition-based diffusion model acceleration sampling method and system

Abstract

The application provides a trajectory decomposition-based diffusion model acceleration sampling method and system based on trajectory decomposition, which comprises the following steps: generating speed change information of a trajectory based on a diffusion model, decomposing the trajectory into a tangential component and a normal component, constructing a time scheduling table offline by statistically accumulating the fluctuation amplitudes of each component; starting a diffusion model sampling process according to an input prompt word, querying the time scheduling table at a current time step to obtain a corresponding step length; when the obtained step length is 1, performing standard denoising calculation by using the original diffusion model; when the step length is greater than 1, using the offline statistical component intensity prior and online direction re-calibration strategy to approximately calculate the state update after skipping the intermediate steps and directly advancing to the target time step; repeating the above querying and calculating steps until the sampling process is completed to obtain the final sampling result. The application realizes online skipping by constructing a time scheduling table offline, significantly reduces model calculation and reduces inference delay.

Description

Technical Field

[0001] This application relates to the field of computer vision, and more specifically, to a diffusion model-based method and system for accelerating sampling based on trajectory decomposition. Background Technology

[0002] While diffusion models excel in generation quality and robustness, the inference phase requires numerous iterative denoising steps, resulting in high end-to-end latency, low throughput, and high energy consumption. Slow inference severely impacts real-world deployments: on one hand, mobile / edge devices have limited computing power and GPU memory; on the other hand, online products have strong requirements for real-time or near-real-time responses. Therefore, accelerating the inference process of diffusion models is a crucial research problem, which can be broken down into two sub-problems: first, when to skip steps; and second, how to approximate the state after skipping steps.

[0003] Regarding these two points, most existing works follow a common paradigm: first, define a measure of "time step similarity," and then formulate a skip-step strategy accordingly. A typical approach is to measure the similarity of model outputs between adjacent or separated time steps, such as differences in noise predictions, velocity, feature representations, or latent space states. When the difference is small enough, the trajectory is considered to be stable and some steps can be skipped. Meanwhile, to make up for the skipped computations, many methods adopt a "reuse" approach: directly using the output of past time steps as a substitute for the current time step output, or performing a linear combination of historical outputs to estimate the new output, thereby reducing redundant computation. However, this "similarity-based reuse" strategy inherently brings challenges: skipping steps introduces approximation errors, which can be amplified or propagated in subsequent steps, causing loss of detail, blurred textures, structural errors, and even semantic shifts.

[0004] Therefore, the focus of the research further shifts to: how to design more reliable output similarity metrics between adjacent steps, how to perform more stable updates after skipping steps, and how to maintain generalization and robustness under different models, samplers, and resolutions.

[0005] A search revealed Chinese patent 202410922398.8, which discloses a method for accelerating diffusion model inference based on model learning. This method employs attention mechanism step pruning and reinforcement learning node matching strategies to achieve acceleration. However, it fails to address the issue of state quality updates after skipping steps and cannot effectively compensate for the omitted evolutionary processes. Summary of the Invention

[0006] To address the shortcomings of existing technologies, the purpose of this application is to provide a diffusion model-based accelerated sampling method and system based on trajectory decomposition.

[0007] A first aspect of this application provides a method for accelerating sampling based on a diffusion model of trajectory decomposition, comprising: Based on the velocity change information of the generated trajectory using the diffusion model, the trajectory is decomposed into tangential and normal components. By statistically analyzing the cumulative fluctuation amplitude of each component, a time schedule table containing the maximum safe step length is constructed offline. The diffusion model sampling process is initiated based on the input prompt word. At the current time step, the time schedule table is queried to obtain the corresponding skip step length. When the obtained step length is 1, the original diffusion model is used for standard denoising calculation; When the step length is greater than 1, the state update after skipping the intermediate steps is approximately calculated by using the prior of component intensity from offline statistics and the online orientation recalibration strategy, and then directly proceeds to the target time step. Repeat the above query and calculation steps until the sampling process is completed and the final sampling result is obtained.

[0008] Optionally, the velocity change information of the trajectory generated based on the diffusion model decomposes the trajectory into tangential and normal components, and constructs an offline time scheduling table containing the maximum safe jump length by statistically analyzing the cumulative fluctuation amplitude of each component, including: The diffusion model is run using the calibration set to obtain velocity change information at each time step, and decomposed into tangential and normal components. The stability prior of the intensity change of the tangential component was obtained statistically. The intensity change of the normal component is stable prior. ; Based on the stability prior and the preset cumulative fluctuation threshold Calculate the function using the step size Calculate the maximum safe step size in the tangential direction respectively. Maximum safe step size in normal direction In the formula, Indicates the step size. Indicates the maximum step size. express ; Take the maximum safe step size in the tangential direction With the maximum safe step size in the normal direction The minimum value is used as the maximum jump interval of the current time step i. ; The time scheduling table is constructed based on the maximum jump interval of each time step.

[0009] Optionally, the step of running the diffusion model using a calibration set to obtain velocity change information at each time step and decomposing it into tangential and normal components includes: The first , The output of the diffusion model at +1 time step is denoted as , ; Define the difference between adjacent time steps Time from Evolved to The step size is recorded as The rate of change within this interval is defined as ;by Treating it as a local reference direction, Decompose into tangential components and normal components : 。

[0010] Optionally, the statistics obtain a stable prior for the intensity change of the tangential component. The intensity change of the normal component is stable prior. ,include: Calculate the intensity of change of the tangential component from the projection relationship. :

[0011] in, Represents the projection operator; The mean of this statistic for all samples at each time step on the calibration set is used to obtain the stable prior for that time step: , Where s represents all samples in the calibration set. Used to depict the first Statistical characteristics of trajectory changes along the tangent near each time step; Define the intensity of normal change :

[0012] The mean of this statistic for all samples at each time step on the calibration set is used to obtain the stable prior for that time step. .

[0013] Optionally, the offline statistical component intensity prior includes: a stable prior including the intensity change of the tangential component. Stable prior for the intensity change of the normal component and step length .

[0014] Optionally, the online orientation recalibration strategy refers to utilizing the outputs of the two adjacent steps before the jump step. and Calculate orthogonal directions ,include: Calculate adjacent differences ; Project the adjacent difference onto The orthogonal subspaces are used to obtain the initial orthogonal components: ; Reorthogonalize the orthogonal components to the current output. The orthogonal directions are obtained by normalization. : .

[0015] Optionally, when the skip step length is greater than 1, the state update after skipping intermediate steps is approximately calculated using the prior component strength obtained from offline statistics and the online orientation recalibration strategy, and the process is directly advanced to the target time step, including: Retrieve the statistical priors for the current time step i from the offline-built time scheduling table, including the stable priors of the intensity change of the tangential component. Stable prior for intensity changes in the normal component and step length ; Calculate the time interval based on the jump length. ,in Let i be the timestamp of step i. The timestamp for the target time step; Get the diffusion model output at the current time step. Calculate its modulus ; Output the two steps immediately preceding the jump step and Calculate orthogonal directions ; By combining offline priors and online states, the output of the target time step is calculated: ; Using the above formula, perform multiple recursive calculations until the timestamp of the target time step is reached. .

[0016] A second aspect of this application provides a diffusion model-based accelerated sampling system based on trajectory decomposition, comprising: The time schedule construction module generates velocity change information of the trajectory based on the diffusion model, decomposes the trajectory into tangential and normal components, and constructs a time schedule containing the maximum safe jump length offline by statistically analyzing the cumulative fluctuation amplitude of each component. The query module initiates the diffusion model sampling process based on the input prompt words, queries the time schedule table at the current time step, and obtains the corresponding jump length. The output module performs standard denoising calculations using the original diffusion model when the obtained step length is 1. When the step length is greater than 1, the state update after skipping the intermediate steps is approximately calculated by using the prior of component intensity from offline statistics and the online orientation recalibration strategy, and then directly proceeds to the target time step. Repeat the above query and calculation steps until the sampling process is completed and the final sampling result is obtained.

[0017] A third aspect of this application provides a method for creating textured images or videos, comprising: Get the text prompt words; Based on any one of the trajectory decomposition-based diffusion model accelerated sampling methods or the trajectory decomposition-based diffusion model accelerated sampling system, the diffusion model sampling process is performed according to the text prompt words; The final sampling result obtained from the sampling process is output as the generated image or video.

[0018] A fourth aspect of this application provides a terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it can be used to execute any of the trajectory decomposition-based diffusion model accelerated sampling methods, textural image or video methods, or to run the trajectory decomposition-based diffusion model accelerated sampling system.

[0019] The accelerated sampling method for diffusion models based on trajectory decomposition provided in this application achieves online skipping by constructing a time schedule offline, significantly reducing model computation and inference latency. It utilizes trajectory decomposition and orientation recalibration strategies to approximate state updates, suppressing error accumulation and ensuring generation quality and semantic consistency. The decoupling of offline and online operations reduces online overhead and improves engineering deployment capabilities.

[0020] Other technical effects resulting from the additional features will be further illustrated in the corresponding embodiments. Attached Figure Description

[0021] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart illustrating an accelerated sampling method based on a trajectory decomposition diffusion model according to an exemplary embodiment; Figure 2 This is a schematic diagram illustrating the framework structure of a diffusion model-based accelerated sampling method according to an exemplary embodiment. Figure 3This is a schematic diagram illustrating the structure of a diffusion model-based accelerated sampling system according to an exemplary embodiment. Detailed Implementation

[0022] The present application will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application, and these all fall within the protection scope of the present application. Parts not described in detail in the following embodiments can be implemented using existing technology.

[0023] Existing methods for accelerating diffusion models face the dual challenges of low reliability in skip-step decisions and the accumulation of state approximation errors after skipping steps. To address these issues, this application provides a trajectory decomposition-based accelerated sampling method for diffusion models.

[0024] Reference Figure 1 and Figure 2 As shown in one embodiment of this application, a diffusion model-based accelerated sampling method based on trajectory decomposition includes the following steps S1-S4.

[0025] S1, Based on the velocity change information of the generated trajectory using the diffusion model, the trajectory is decomposed into tangential and normal components. By statistically analyzing the cumulative fluctuation amplitude of each component, a time schedule table containing the maximum safe step length is constructed offline. Specifically, the diffusion model generation trajectory refers to the continuous change path formed by the output state at each time step in the data space during the process of the diffusion model iteratively evolving from the initial noise to the final clear image.

[0026] Its velocity change information refers to the rate of change vector of the output state between adjacent time steps (i.e., the state increment per unit time), as well as the amplitude and direction fluctuation characteristics of this vector during the trajectory evolution process.

[0027] Specifically, step S1 belongs to the offline statistics stage.

[0028] S2, start the diffusion model sampling process based on the input prompt word, and query the time schedule table at the current time step to obtain the corresponding skip step length; Specifically, the reasoning phase begins in step S2.

[0029] S3, when the obtained step length is 1, the original diffusion model is used for standard denoising calculation; When the step length is greater than 1, the state update after skipping intermediate steps is approximately calculated by using the prior of component intensity from offline statistics and the online orientation recalibration strategy, and then directly proceeds to the target time step. S4. Repeat the above query and calculation steps until the sampling process is completed and the final sampling result is obtained.

[0030] The method described in the above embodiment differs from existing advanced diffusion cache methods such as TeaCache and TaylorSeer. For example, TeaCache and TaylorSeer directly use the relative changes in output features between adjacent steps as the evaluation criterion. When values, such as those with small changes only along the normal direction, occur, they often fail to be recognized. However, the change in direction is extremely important for the generation of the entire diffusion process. Therefore, the above embodiment decouples the tangential and normal directions into finer tangential and normal directions, constructing a more stable criterion. Furthermore, the reconstruction performed during step skipping in this embodiment differs from the reuse of historical outputs or residuals in TeaCache, MagCache, and TaylorSeer. Reconstruction using offline information can more accurately approximate the output of that step during step skipping, ensuring output accuracy.

[0031] It should be noted that the step length in this embodiment... It is an integer greater than or equal to 1.

[0032] Step length A value of 1 indicates that no skipping acceleration is performed at the current time step; the sampling process proceeds step by step according to the standard sampling procedure. This is the baseline mode to ensure that the sampling process can be completed normally. Skip step size. A value greater than 1 indicates that the current time step meets the stability condition, and a skip step acceleration is performed, skipping the intermediate steps. -Calculation of 1 time step.

[0033] When building the time schedule offline, if the theoretical safe step size calculated based on the cumulative fluctuation amplitude is less than 1, it is forcibly set to 1. This constraint ensures that the sampling process always moves forward, avoiding infinite loops caused by a step size of 0 or logical errors caused by a negative step size, thus guaranteeing the robustness and engineering usability of the acceleration method.

[0034] Existing methods often rely on the "surface similarity" of outputs from adjacent time steps to determine skip steps, which can easily lead to misjudgments in intervals with drastic structural changes or semantic rearrangements, causing errors to accumulate and amplify in subsequent steps. This application introduces a more robust criterion and constraint mechanism (specifically, it approaches the issue from two dimensions: tangential and normal changes, comparing them with thresholds for more nuanced judgment). In some embodiments of this application, S1, based on the velocity change information of the generated trajectory using a diffusion model, the trajectory is decomposed into tangential and normal components. By statistically analyzing the cumulative fluctuation amplitude of each component, a time schedule table containing the maximum safe skip step length is constructed offline. This can be achieved through the following steps. S101, such as Figure 2As shown, the diffusion model is run using the calibration set to obtain velocity change information at each time step, and decomposed into tangential and normal components.

[0035] Specifically, the first The model output at each time step is denoted as... (By the output of the (i-1)th time step) Samples were obtained by sending samples into the sampler. (and then obtained through model inference), the first The model output at each time step is denoted as... To describe the local increment of time step evolution, the difference between adjacent time steps is defined. Meanwhile, time from Evolved to The step size is recorded as Therefore, the rate of change within this interval can be written as: To more precisely characterize the direction of change, Treating it as a local reference direction, Decompose into tangential components and normal components : (1) S102, statistically obtained stable priors for the intensity changes of the tangential and normal components. and .

[0036] Specifically, regarding the intensity of change in the tangential component It can be calculated from the projection relationship:

[0037] Furthermore, by averaging this statistic across all samples at each time step on the calibration set, the stable prior for that time step is obtained: , Where s represents all samples in the calibration set. Used to depict the first Statistical characteristics of the trajectory changes along the tangent near each time step.

[0038] Specifically, during the offline phase, the intensity of normal change is defined as follows:

[0039] We use the information of the normal component to calculate the data and use it as a key quantity in the subsequent statistical process.

[0040] Similar to the tangential component, the statistic is averaged across all samples at each time step on the calibration set to obtain the stable prior for that time step:

[0041] S103, based on stable priors and a preset cumulative fluctuation threshold Calculate the function using the step size ; Calculate the maximum safe step size in the tangential direction respectively. and the maximum safe step size in the normal direction In the formula, Represents the set of integers. Indicates step size, Indicates the maximum step size. express ; S104, Take the maximum safe step size in the tangential direction. Maximum safe step size in normal direction The minimum value is used as the maximum jump interval of the current time step. Based on this, a time scheduling table is constructed.

[0042] The above embodiments employ an offline step skip table construction technique based on trajectory decomposition. By accurately identifying compressible, smooth intervals and critical intervals requiring careful calculation based on the cumulative fluctuation amplitude of components, more reliable step skip timing and step size selection are achieved. This not only solves the technical problems of existing methods that are prone to misjudgment and unstable step skipping due to relying solely on the similarity of time step output, but also maintains stable acceleration gains under different prompts, resolutions, and content complexities, and provides adjustable control between quality and speed through threshold settings.

[0043] Skipping steps is not just about "omitting a few steps," but more importantly, about how to compensate for the omitted evolutionary process. Simply reusing historical outputs or linear extrapolation often fails to characterize the nonlinear changes in the trajectory, potentially leading to missing details, texture collapse, or semantic drift. This application addresses the estimation of intermediate states after skipping steps by designing an update method that better fits the dynamics of the diffusion trajectory. In some specific embodiments of this application, when the skip step length is greater than 1 in S3, the state update after skipping intermediate steps is approximated using offline statistical component intensity priors and an online orientation recalibration strategy, directly advancing to the target time step. This can be achieved through the following steps: S301, obtain offline prior parameters.

[0044] Retrieve the statistical priors for the current time step i from the offline-built time schedule table, including the tangential strength prior. Prior normal strength and step length ; S302, calculate the time span.

[0045] Calculate the time interval based on the jump length. ,in Let i be the timestamp of step i. The timestamp for the target time step; S303, Get the current online status.

[0046] Get the model output at the current time step Calculate its modulus .

[0047] S304 performs online calculations and orientation recalibration.

[0048] Output the two steps immediately preceding the jump step and Calculate orthogonal directions .

[0049] S305 combines offline priors and online states to calculate the output of the target time step: Perform multiple recursive calculations until the timestamp of the target time step is reached. Thus, we can obtain the following: Figure 2 shown .

[0050] In one specific implementation, the derivation process of the above approximation formula is based on the principle of trajectory decomposition, as follows: From the trajectory decomposition formula, we can see that the single-step evolution rate It can be decomposed into tangential and normal components:

[0051] Therefore, a single-step update can be denoted as:

[0052] when With a step size of When the evolution is stable within a certain interval, the tangential component can be used. and normal components Approximate updates are performed. To obtain stable update results, this application consists of two parts: a tangential component scheduling strategy and a direction correction strategy.

[0053] Tangential component scheduling strategy, including tangential component offline statistics and tangential update strategy.

[0054] Offline statistics of tangential components, i.e., obtaining stable priors of tangential components at time steps. .

[0055] Tangential update strategy: within the jump interval, adopt Update the tangential component, where This is an unknown term. Therefore, the prior obtained from offline statistics is used. Approximate substitution.

[0056] Due to the intensity of the tangential component of different samples Within the same time step, relatively stable statistical patterns are often observed. Therefore, introducing offline priors can maintain scale consistency during step-by-step progression and alleviate excessive smoothing or over-sharpening caused by extrapolation errors.

[0057] The direction correction strategy includes offline statistics of normal components, normal component update mechanism, and direction recalibration strategy.

[0058] Offline statistics of the normal components yield stable priors for the normal components. .

[0059] Normal component update mechanism: Normal components Deconstructing it into intensity term × direction term, it can be expressed as:

[0060] in Using offline priors To approximate, Obtained directly from the current output of online inference. (Final direction term) By using a direction recalibration strategy, the direction of normal update is aligned with the evolution trend of the current sample, thereby improving the stability and consistency after the jump step.

[0061] The direction recalibration strategy is as follows: Initial evolutionary trend capture: Before entering the jump interval, utilize the output of the two adjacent steps before the jump. and This is used to capture local evolutionary trends and generate initial orthogonal direction references based on them. First, the adjacent difference is defined:

[0062] Then Projected to The orthogonal subspaces are used to obtain the initial orthogonal components:

[0063] This quantity characterizes the change from... During the evolution process, the deflection trend relative to the main direction at the previous moment can be used as the initial direction reference for the jump segment.

[0064] Dynamic direction normalization: Within the jump interval, in order to obtain "the current time" Matching orthogonal directions and ensuring they are aligned with Strictly orthogonal, using the orthogonal reference from the previous time step. Reorthogonalize to the current output Above, and then unitize:

[0065] Through the normalization process of "stepwise orthogonalization", the following is obtained: On the one hand, it inherits the historical trend of deviation; on the other hand, it always meets the current needs. The orthogonal constraints significantly reduce the risk of directional drift during the jump phase. Finally, the update of the normal component is obtained.

[0066] The above embodiments employ a decomposition and reconstruction approximation technique using tangential and normal components, which can achieve more accurate estimation of the output after skipping steps and lower error accumulation, solving the technical problems of existing skipping schemes that directly reuse historical outputs or perform coarse extrapolation, resulting in missing details, structural distortion, and semantic drift. The above embodiments employ online orientation re-determination technology, which can achieve stable estimation and continuous correction of orthogonal orientation during the skip-step phase, and solve the technical problems of orientation drift, error propagation, and insufficient robustness in complex scenarios introduced by the skip-step approximation. The above embodiments employ an inference acceleration framework technology that decouples offline statistics from online lightweight computing. This technology can achieve significant inference acceleration and stronger engineering deployability with lower online overhead, solving the technical problems of heavy online computing burden, complex integration, and difficulty in cross-task migration of existing acceleration methods.

[0067] Based on the same technical concept, other embodiments of this application provide a diffusion model-based accelerated sampling system 100 based on trajectory decomposition, such as... Figure 3 As shown, it includes: The time schedule construction module 110 generates velocity change information of the trajectory based on the diffusion model, decomposes the trajectory into tangential and normal components, and constructs a time schedule containing the maximum safe jump length offline by statistically analyzing the cumulative fluctuation amplitude of each component. The query module 120 initiates the diffusion model sampling process based on the input prompt words, and queries the time schedule table at the current time step to obtain the corresponding jump length; Output module 130, when the obtained step length is 1, uses the original diffusion model for standard denoising calculation; when the step length is greater than 1, it uses the offline statistical component intensity prior and online orientation recalibration strategy to approximate the state update after skipping the intermediate steps and directly advances to the target time step. Repeat module 140, repeating the above query and calculation steps until the sampling process is completed and the final sampling result is obtained.

[0068] The specific implementation techniques of each module / unit in the above examples of this application can be referred to the steps of the accelerated sampling method based on trajectory decomposition based on the diffusion model in the above embodiments, and will not be repeated here.

[0069] Based on the same technical concept, other embodiments of this application provide a method for generating text images or videos, including the following steps: S10, retrieve text prompts; S20, based on any one of the trajectory decomposition-based diffusion model accelerated sampling methods or trajectory decomposition-based diffusion model accelerated sampling systems, the diffusion model sampling process is executed according to the text prompt words; In this step, you can follow steps S21-24 as follows: S21, Based on the velocity change information of the generated trajectory using the diffusion model, the trajectory is decomposed into tangential and normal components. By statistically analyzing the cumulative fluctuation amplitude of each component, a time schedule table containing the maximum safe step length is constructed offline. S22, start the diffusion model sampling process according to the input prompt word, and query the time schedule table at the current time step to obtain the corresponding skip step length; S23, when the obtained step length is 1, the original diffusion model is used for standard denoising calculation; when the step length is greater than 1, the offline statistical component intensity prior and online orientation recalibration strategy are used to approximate the state update after skipping the intermediate steps and directly advance to the target time step. S24. Repeat the above query and calculation steps until the sampling process is completed and the final sampling result is obtained.

[0070] The specific implementation of the above steps can be referred to the preferred implementation in the foregoing embodiments, and will not be repeated here.

[0071] S30 outputs the final sampling result obtained from the sampling process as the generated image or video.

[0072] The above embodiments achieve efficient generation of text-based images or videos, significantly reducing inference latency while ensuring generation quality, and improving response speed and user experience in practical application scenarios.

[0073] The preferred features in the above embodiments can be used individually in any embodiment, or in any combination thereof, provided they do not conflict with each other. Furthermore, parts not described in detail in the embodiments can be implemented using existing technologies.

[0074] The following examples and comparative examples will be used to further illustrate this application in order to better understand the above-mentioned technical solutions. It should be understood that the following are only some examples and are not intended to limit this application.

[0075] Example 1: Accelerated Sampling via Diffusion in Text-to-Image Tasks 1. Input Data and Inference Configuration (1) The input is a text prompt, which can be real user input or text from a public dataset, such as GenEval.

[0076] (2) Existing mainstream text graph models such as BAGEL and FLUX can be used for diffusion models. Euler and UniPC can be used as samplers, and a unified noise scheduling and total number of steps should be set. .

[0077] (3) Output is the generated image And the output of the model at each time step is recorded as .

[0078] 2. Implementation Steps Step 1: Offline Stage Statistics and Skip-Step Table Construction (1) Using a smaller prompt as the calibration set, standard inference is run to obtain the output sequence at each time step. Furthermore, trajectory decomposition is performed on adjacent differences to obtain parallel terms. Orthogonal terms : (2) Statistical analysis of offline prior quantities: tangential vector strength With normal update strength Based on this, an offline skip step table is constructed. .

[0079] (3) It serves as an offline calibration result storage for online inference and querying.

[0080] Step 2: Online Stage Skip Reasoning and Output Approximation (1) Perform inference on the input prompt at time step Query the skip step table: If Then update according to standard reasoning; if This triggers a jump step and crosses the subsequent... An intermediate time step.

[0081] (2) Within the skip interval, the output is reconstructed using the skip approximation update formula:

[0082] in Statistics from offline sources From the current online output amplitude, Obtained by the orientation recalibration strategy.

[0083] (3) Complete the reasoning process according to the time schedule and generate the final image.

[0084] 3. Experimental Results To verify the effectiveness of this application in text-based image tasks, the generation quality and inference overhead can be compared between "baseline non-skipping" and "skipping acceleration using this invention" under fixed generation quantity and resolution settings: (1) Improved inference efficiency: Under the same hardware and resolution, this application can reduce the number of effective sampling steps or reduce redundant computation per step, thereby significantly shortening the end-to-end inference time. It achieves 4x speedup on the FLUX.1-dev model and 4x speedup on BAGEL.

[0085] (2) Quality preservation: Under the same prompt conditions, the generated results are close to the baseline in terms of main structure, semantic consistency and texture details, and are less prone to blurring, structural breakage or semantic drift caused by skipping steps.

[0086] (3) Mass-velocity balance: Compared with other methods, this application achieves less mass loss with the same speed increase, or achieves higher acceleration with the same mass.

[0087] Example 2: Diffusion-accelerated sampling in text-to-video tasks 1. Input Data and Inference Configuration (1) The input is a text prompt word, which may include descriptions of the subject, actions, and camera language.

[0088] (2) The diffusion video generation model adopts the existing mainstream text-based video diffusion model Wan2.1-1.3B, and sets the output frame count, resolution, sampler and total number of steps. .

[0089] (3) The output of the model at each time step is denoted as The output video is .

[0090] 2. Implementation Steps Step 1: Offline stage calibration (1) Select a representative set of video cue words in InternVid-10M-FLT and record the output trajectory under standard inference. .

[0091] (2) To Execution trajectory decomposition and statistics are obtained Build an offline skipping table .

[0092] (3) Deploy the offline table to the online inference module.

[0093] Step 2: Online Phase Skip-Step Reasoning (1) Query by time step It performs skip steps in stable regions and forward inference in sensitive regions.

[0094] (2) Reconstruction based on approximate update formula during the skip step stage And obtain through direction reorientation .

[0095] (3) After completing the reasoning, decode to generate video output.

[0096] 3. Experimental Results To verify the generalization ability of this application on textural video tasks, a comparison was made with a baseline scheme that does not adopt this application and existing acceleration schemes under a fixed frame rate and resolution configuration. Experimental results show that: (1) Significant speed improvement: Without changing the backbone network structure and training method of the diffusion model, this application reduces redundant time step calculations through an offline skip table-driven skip strategy, thereby effectively reducing inference time and improving generation throughput. Compared with the comparison method, this application can achieve an inference speedup of up to 2×.

[0097] (2) Achieving the best overall balance between speed and quality: This application achieves the fastest inference speed while maintaining a leading level of generation quality. Using VBench as the evaluation index, this application achieves the best score among the comparative schemes, indicating that it can effectively suppress the quality loss caused by skipping steps while improving inference efficiency, thus achieving the best balance between speed and quality.

[0098] (3) Best visualization effect: Among the comparison schemes, the video generated by this application has the best visual effect, with more stable cross-frame content consistency and motion continuity, and significantly reduced phenomena such as flickering, drifting, shaking and discontinuous motion; in scenarios with large displacement, rapid camera switching, occlusion and drastic structural changes, the orientation resetting mechanism can continuously correct the jump direction, making the update more in line with the real trajectory, thereby reducing the probability of structural breakage, main body deformation and detail collapse.

[0099] Based on the same technical concept, in other embodiments of this application, a terminal is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it can be used to execute the above-mentioned method for accelerating diffusion model inference based on trajectory decomposition, the method for generating images or videos, or to run the above-mentioned system for accelerating diffusion model inference based on trajectory decomposition.

[0100] Optionally, the memory is used to store programs; the memory may include volatile memory, such as random-access memory (RAM), such as static random-access memory (SRAM), double data rate synchronous dynamic random-access memory (DDR SDRAM), etc.; the memory may also include non-volatile memory, such as flash memory. The memory is used to store computer programs (such as application programs and functional modules that implement the above methods), computer instructions, etc., and the aforementioned computer programs and computer instructions can be partitioned and stored in one or more memories. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by the processor.

[0101] The aforementioned computer programs, computer instructions, etc., can be stored in partitions within one or more memory locations. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by a processor.

[0102] A processor is used to execute a computer program stored in memory to implement the various steps of the methods involved in the above embodiments. For details, please refer to the relevant descriptions in the preceding method embodiments.

[0103] The processor and memory can be separate structures or integrated structures. When the processor and memory are separate structures, they can be coupled together via a bus.

[0104] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0105] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0106] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0107] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0108] The foregoing has described some specific embodiments of this application. It should be understood that this application is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the substantive content of this application. The above-described preferred features can be used in any combination without conflict.

Claims

1. A diffusion model-based accelerated sampling method based on trajectory decomposition, characterized in that, include: Based on the velocity change information of the generated trajectory using the diffusion model, the trajectory is decomposed into tangential and normal components. By statistically analyzing the cumulative fluctuation amplitude of each component, a time schedule table containing the maximum safe step length is constructed offline. The diffusion model sampling process is initiated based on the input prompt word. At the current time step, the time schedule table is queried to obtain the corresponding skip step length. When the obtained step length is 1, the original diffusion model is used for standard denoising calculation; When the step length is greater than 1, the state update after skipping the intermediate steps is approximately calculated by using the prior of component intensity from offline statistics and the online orientation recalibration strategy, and then directly proceeds to the target time step. Repeat the above query and calculation steps until the sampling process is completed and the final sampling result is obtained.

2. The accelerated sampling method based on trajectory decomposition using a diffusion model according to claim 1, characterized in that, The velocity change information of the trajectory generated based on the diffusion model decomposes the trajectory into tangential and normal components. By statistically analyzing the cumulative fluctuation amplitude of each component, an offline time scheduling table containing the maximum safe jump length is constructed, including: The diffusion model is run using the calibration set to obtain velocity change information at each time step, and decomposed into tangential and normal components. The stability prior of the intensity change of the tangential component was obtained statistically. The intensity change of the normal component is stable prior. ; Based on the stability prior and the preset cumulative fluctuation threshold Calculate the function using the step size Calculate the maximum safe step size in the tangential direction respectively. Maximum safe step size in normal direction In the formula, Indicates step size, Indicates the maximum step size. express ; Take the maximum safe step size in the tangential direction With the maximum safe step size in the normal direction The minimum value is used as the maximum jump interval of the current time step i. ; The time scheduling table is constructed based on the maximum jump interval of each time step.

3. The accelerated sampling method based on trajectory decomposition using a diffusion model according to claim 2, characterized in that, The process involves running a diffusion model using a calibration set to obtain velocity change information at each time step, and decomposing it into tangential and normal components, including: The first , The output of the diffusion model at +1 time step is denoted as , ; Define the difference between adjacent time steps Time from Evolved to The step size is recorded as The rate of change within this interval is defined as ;by Treating it as a local reference direction, Decompose into tangential components and normal components : 。 4. The accelerated sampling method based on trajectory decomposition using a diffusion model according to claim 2, characterized in that, The statistics yielded a stable priori information regarding the intensity change of the tangential component. The intensity change of the normal component is stable prior. ,include: Calculate the intensity of change of the tangential component from the projection relationship. : ； in, Represents the projection operator; The mean of this statistic for all samples at each time step on the calibration set is used to obtain the stable prior for that time step: ， Where s represents all samples in the calibration set. Used to depict the first Statistical characteristics of trajectory changes along the tangent near each time step; Define the intensity of normal change : ； The mean of this statistic for all samples at each time step on the calibration set is used to obtain the stable prior for that time step. .

5. The accelerated sampling method based on trajectory decomposition using a diffusion model according to claim 1, characterized in that, The prior for component intensity in the offline statistics includes: a stable prior for the intensity change of the tangential component. Stable prior for the intensity change of the normal component and step length .

6. The accelerated sampling method based on trajectory decomposition using a diffusion model according to claim 1, characterized in that, The online orientation recalibration strategy refers to utilizing the outputs of the two adjacent steps before the jump step. and Calculate orthogonal directions ,include: Calculate adjacent differences ; Project the adjacent difference onto The orthogonal subspaces are used to obtain the initial orthogonal components: ； Reorthogonalize the orthogonal components to the current output. The orthogonal directions are obtained by normalization. : 。 7. The accelerated sampling method based on trajectory decomposition using a diffusion model according to claim 1, characterized in that, When the skip step length is greater than 1, the state update after skipping intermediate steps is approximately calculated using the prior component strength obtained from offline statistics and the online orientation recalibration strategy, and then directly advanced to the target time step, including: Retrieve the statistical priors for the current time step i from the offline-built time scheduling table, including the stable priors of the intensity change of the tangential component. Stable prior for intensity changes in the normal component and step length ; Calculate the time interval based on the jump length. ,in Let i be the timestamp of step i. The timestamp for the target time step; Get the diffusion model output at the current time step. Calculate its modulus ; Output the two steps immediately preceding the jump step and Calculate orthogonal directions ; By combining offline priors and online states, the output of the target time step is calculated: ； Using the above formula, perform multiple recursive calculations until the timestamp of the target time step is reached. .

8. A diffusion model-based accelerated sampling system based on trajectory decomposition, characterized in that, include: The time schedule construction module generates velocity change information of the trajectory based on the diffusion model, decomposes the trajectory into tangential and normal components, and constructs a time schedule containing the maximum safe jump length offline by statistically analyzing the cumulative fluctuation amplitude of each component. The query module initiates the diffusion model sampling process based on the input prompt words, queries the time schedule table at the current time step, and obtains the corresponding jump length. The output module performs standard denoising calculations using the original diffusion model when the obtained step length is 1. When the step length is greater than 1, the state update after skipping the intermediate steps is approximately calculated by using the prior of component intensity from offline statistics and the online orientation recalibration strategy, and then directly proceeds to the target time step. Repeat the above query and calculation steps until the sampling process is completed and the final sampling result is obtained.

9. A method for generating images or videos, characterized in that, include: Get the text prompt words; Based on the diffusion model accelerated sampling method based on trajectory decomposition according to any one of claims 1 to 7 or the diffusion model accelerated sampling system based on trajectory decomposition according to claim 8, the diffusion model sampling process is performed according to the text prompt words; The final sampling result obtained from the sampling process is output as the generated image or video.

10. A terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it can be used to perform the method of any one of claims 1-7 and 9, or to run the system of claim 8.

Images