A diffusion model inference acceleration method, device and storage medium

Abstract

The application discloses a diffusion model reasoning acceleration method and device and a storage medium, and belongs to the technical field of artificial intelligence. The method comprises the following steps: performing reverse denoising reasoning through a target diffusion model to generate a target image; in the process, an acceleration mechanism is enabled, which comprises the following steps: determining a target number of key time steps based on a non-uniform time step sequence; determining a main key time step set from the target number of key time steps based on a preset early composition protection parameter; performing full-amount forward calculation according to the target number of key time steps and the main key time step set by means of an encoder and a decoder of the target diffusion model and in combination with an asymmetric caching strategy, respectively generating corresponding target features, and updating cached features into the target features; and multiplexing the cached target features at non-key time steps by means of the encoder and the decoder. The technical scheme provided by the application can improve the reasoning speed of the diffusion model and ensure that the image quality generated by the diffusion model has no obvious loss.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence deep learning technology, and in particular to a method, device and storage medium for accelerating diffusion model inference. Background Technology

[0002] In scenarios such as image generation, video generation, 3D content generation, and image editing, diffusion models can be used to generate corresponding content, thereby improving generation quality and text alignment capabilities. However, diffusion models suffer from problems such as inference latency and high memory usage: the generation process of diffusion models relies on serial iterative denoising, and native inference requires tens to hundreds of model forward propagations, resulting in high inference latency; furthermore, in the pursuit of higher generation quality, the number of model parameters continues to increase.

[0003] Currently, research on the inference efficiency of diffusion models mainly focuses on four directions: sampling strategy optimization, model quantization compression, training-free architecture optimization, and model distillation and pruning. However, each approach has varying degrees of technical limitations: Sampling strategy optimization has a physical limit to the number of steps in the few-step sampler compression; excessive compression leads to a decrease in generation quality. Sampling scheduling optimization cannot reduce the computational cost and memory usage of a single inference step. Model quantization compression reduces memory usage and improves computational efficiency by lowering the numerical precision of model weights and activations; however, simultaneous 4-bit quantization of weights and activations can easily cause loss of high-frequency details and textures in the generated image, reducing the generation effect of complex text prompts. Training-free architecture optimization reduces the computational cost per step by pruning and merging redundant tokens. And Aggressive token merging can lead to blurred generated subjects and reduced text alignment capabilities; furthermore, this approach only optimizes the computational cost of the attention module and cannot reduce the model's memory usage. Model distillation and pruning reduce the number of model parameters and computational cost through model structure pruning and knowledge distillation; however, model pruning and distillation require a massive number of high-end computational examples, resulting in extremely high training costs.

[0004] In summary, existing efficiency optimization schemes for diffusion models cannot simultaneously meet multiple requirements such as no training / light training, high generation quality preservation, and low computational and memory overhead; therefore, a more reliable solution is needed. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a diffusion model inference acceleration method, device and storage medium that can improve the inference speed of diffusion models while ensuring that the quality of the generated images is not significantly lost.

[0006] To achieve the above objectives, the present invention employs the following technical solution:

[0007] On one hand, the present invention provides a method for accelerating diffusion model inference, the method comprising:

[0008] The text description information is input into the target diffusion model, and the target image is generated by reverse denoising inference through the target diffusion model;

[0009] The acceleration mechanism enabled during the reverse denoising inference process using the target diffusion model includes:

[0010] Based on the non-uniform time step sequence, the target number of key time steps are determined; the non-uniform time step sequence is obtained by solving the time optimization objective function, which is constructed based on the truncation error generated by the ODE solver of the target diffusion model; the key time steps are the time steps where the local truncation error is greater than or equal to a preset threshold.

[0011] Based on preset early-stage mapping protection parameters, a set of main key time steps is determined from the target number of key time steps;

[0012] The encoder and decoder of the target diffusion model generate corresponding target features by performing full forward computation based on the target number of key time steps and the set of main key time steps, combined with an asymmetric caching strategy, and update the cached features to the target features.

[0013] And the target features cached are reused at non-critical time steps by the encoder and the decoder.

[0014] In some possible implementations, the time-step optimization objective function is constructed using the following method:

[0015] Based on the truncation error generated by the ODE solver of the target diffusion model, the initial time step optimization objective function is constructed as shown in equation (1):

[0016] (1);

[0017] The constraints for the initial time step optimization objective function are as follows: ;

[0018] in, Indicates that ODE is at time step The true solution, This indicates that the ODE solver is at time step The approximate solution is obtained, and the truncation error generated by the ODE solver is determined based on the approximate solution and the true solution. Indicates the total number of sampling steps; Indicates the first Each time step Indicates the first Each time step 0 represents the initial time step, and 0 represents the final time step;

[0019] Based on the initial time step optimization objective function, the error upper bound inequality between the true solution and the approximate solution is constructed, as shown in equation (2):

[0020] (2);

[0021] in, This represents the approximate solution of the ODE solver at the final time step. This represents the true solution of ODE at the final time step; This represents the noise standard deviation at the final time step. The noise standard deviation represents the initial time step; This represents the half-log signal-to-noise ratio at the final time step. This represents the semi-log signal-to-noise ratio at the initial time step; This represents a constant related to the order of the ODE solver; Indicates time step Local truncation error; This represents the multi-step weighting coefficients of the ODE solver. This represents the global index of the current time step. Indicates the order of the ODE solver. This represents the history offset index of the ODE solver;

[0022] Based on the error upper bound inequality, the time step optimization objective function is determined as shown in equation (3):

[0023] (3);

[0024] The constraints of the time-step optimization objective function are as follows: ;

[0025] in, This represents the first time step in a non-uniform time step sequence. The corresponding semi-log signal-to-noise ratio, In a non-uniform time step sequence, the th time step The corresponding semi-log signal-to-noise ratio, In a non-uniform time step sequence, the th time step The corresponding semi-log signal-to-noise ratio, In a non-uniform time step sequence, the th time step The corresponding semi-logarithmic signal-to-noise ratio.

[0026] In some possible implementations, determining the target number of key time steps based on a non-uniform time step sequence includes:

[0027] Based on the local truncation error of each non-uniform time step, the non-uniform time step sequence is filtered to determine the set of candidate key time steps, as shown in equation (4):

[0028] (4);

[0029] in, Represents the set of candidate key time steps; Indicates non-uniform time steps Local truncation error; This represents the maximum value of the local truncation error for all non-uniform time steps; This represents the filtering threshold coefficient; Represents the total number of non-uniform time intervals;

[0030] From the set of candidate key time steps, retain the target number of key time steps; wherein, the target number of key time steps is as shown in equation (5):

[0031] (5);

[0032] in, Indicates the number of targets at key time steps. This represents the global retention ratio. This indicates the total number of sampling steps.

[0033] In some possible implementations, the target features include high-level semantic structure features corresponding to the encoder and intermediate high-level texture features corresponding to the decoder;

[0034] The encoder and decoder of the target diffusion model generate corresponding target features based on the target number of key time steps and the set of main key time steps, and perform full forward computation in conjunction with an asymmetric caching strategy, and update the cached features with the target features, including:

[0035] The encoder performs full forward computation at each of the major key time steps in the set of major key time steps to generate high-level semantic structure features for the corresponding key time steps, and updates the cached features with the high-level semantic structure features.

[0036] After performing full forward computation at each major key time step, the encoder performs full forward computation at non-major key time steps that meet preset conditions in the target number of key time steps to generate high-level semantic structure features for the corresponding key time steps, and updates the cached features to the high-level semantic structure features.

[0037] The decoder performs full forward computation at each of the target number of key time steps to generate intermediate high-level texture features for the corresponding key time steps, and updates the cached features with the intermediate high-level texture features.

[0038] The reuse of cached target features by the encoder and the decoder at non-critical time steps includes:

[0039] The encoder reuses the high-level semantic structure features of the cache at the non-critical time steps;

[0040] The decoder reuses the intermediate high-level texture features of the cache at the non-critical time step.

[0041] In some possible implementations, performing full forward computation through the encoder to generate high-level semantic structure features corresponding to key time steps, and updating the cached features to the high-level semantic structure features includes:

[0042] Iterate through each non-uniform time step in the non-uniform time step sequence;

[0043] When traversing to any non-uniform time step, based on the global time step mapping table, it is determined whether the current non-uniform time step is a critical time step; the global time step mapping table is determined according to the non-uniform time step sequence and the target number of critical time steps, and the global time step mapping table includes the value of each non-uniform time step and the value representing whether the corresponding non-uniform time step is a critical time step.

[0044] If the current non-uniform time step is a critical time step, the sequence number of the current critical time step is obtained based on the global critical step mapping table; the global critical step mapping table is generated according to the target number of critical time steps.

[0045] Based on the sequence number of the current critical time step, determine whether the current critical time step belongs to the set of main critical time steps;

[0046] When the current key time step belongs to the set of main key time steps, the encoder performs full forward computation at the current key time step to generate high-level semantic structure features for the corresponding key time step, and updates the cached features to the high-level semantic structure features.

[0047] If the current critical time step does not belong to the set of main critical time steps, determine whether the sequence number of the current critical time step is an integer multiple of the preset skip step parameter;

[0048] When the sequence number of the current key time step is an integer multiple of the preset skip step parameter, the encoder performs full forward computation at the current key time step to generate high-level semantic structure features corresponding to the key time step, and updates the cached features to the high-level semantic structure features.

[0049] In some possible implementations, performing full forward computation through the decoder to generate intermediate high-level texture features corresponding to key time steps, and updating the cached features to the intermediate high-level texture features, includes:

[0050] When the current non-uniform time step is a critical time step, the decoder performs full forward computation at the current critical time step to generate intermediate high-level texture features corresponding to the critical time step, and updates the cached features to the intermediate high-level texture features.

[0051] In some possible implementations, the method further includes:

[0052] The cache reuse interval corresponding to each key time step is determined as shown in Equation (6):

[0053] (6);

[0054] in, Indicates the first The cache reuse interval corresponding to each key time step; Indicates the cache interval adjustment factor; Indicates the first Key time steps and the first The time interval between two adjacent key time steps; Indicates the first The noise standard deviation of each key time step;

[0055] For each critical time step, the encoder and decoder reuse the target features cached in the current critical time step in the non-critical time step corresponding to the cache reuse interval of the current critical time step;

[0056] If there are non-critical time steps after the cache reuse interval of the current critical time step, the encoder and the decoder perform full forward computation at the first non-critical time step after the cache reuse interval to generate corresponding target features, and update the cached features to the target features.

[0057] The encoder and decoder reuse the cached target features in non-critical time steps after the first non-critical time step until the next critical time step. Then, the encoder and decoder perform full forward computation to generate the corresponding target features and update the cached features to the target features.

[0058] In some possible implementations, the set of main key time steps is determined from the target number of key time steps based on preset early-stage mapping protection parameters, as shown in equation (7):

[0059] (7);

[0060] in, This represents the set of key time steps. This indicates the number of key time steps in the target sequence. A key time step; This indicates the preset parameters for protecting the initial layout. This represents the total number of critical time steps.

[0061] On the other hand, an electronic device is provided, the device including a processor and a memory, the memory storing at least one instruction and at least one program, the at least one instruction and the at least one program being loaded and executed by the processor to implement the diffusion model inference acceleration method as described above.

[0062] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction and at least one program are stored therein, the at least one instruction and the at least one program being loaded and executed by a processor to implement the diffusion model inference acceleration method as described above.

[0063] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0064] In this invention, by inputting textual description information into the target diffusion model, and performing reverse denoising inference through the target diffusion model, a target image is generated. Enabling an acceleration mechanism during the reverse denoising inference process through the target diffusion model can improve the inference speed of the diffusion model while maximizing the generation quality.

[0065] The acceleration mechanism implemented during the reverse denoising inference process using the target diffusion model includes: determining the target number of key time steps based on a non-uniform time step sequence; the non-uniform time step sequence is obtained by solving a time optimization objective function, which is constructed based on the truncation error generated by the ODE solver of the target diffusion model, and key time steps are those where the local truncation error is greater than or equal to a preset threshold; determining the main key time step set from the target number of key time steps based on preset early-stage graph protection parameters, thereby avoiding the loss of graph information in the early stages of diffusion and thus significantly improving the robustness of generation and semantic alignment accuracy in high-speed-ratio scenarios; generating corresponding target features by performing full forward computation based on the target number of key time steps and the main key time step set, combined with an asymmetric caching strategy, using the encoder and decoder of the target diffusion model, and updating the cached features to target features; and reusing cached target features in non-key time steps by the encoder and decoder, which can balance structural fidelity and detail generation, reduce the number of executions, improve acceleration efficiency, and thus balance generation quality and acceleration effect. Attached Figure Description

[0066] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0067] Figure 1 This is a flowchart illustrating the process of enabling an acceleration mechanism during reverse denoising inference using the target diffusion model, as provided in an embodiment of the present invention.

[0068] Figure 2 This is a schematic diagram of the structure of a diffusion model inference acceleration device provided in an embodiment of the present invention. Detailed Implementation

[0069] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0070] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0071] In this embodiment of the invention, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0072] Various exemplary embodiments, features, and aspects of the present invention will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0073] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0074] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0075] Furthermore, to better illustrate the present invention, numerous specific details are set forth in the following detailed embodiments. Those skilled in the art will understand that the present invention can be practiced without certain specific details. In some instances, methods, means, elements, and circuits well known to those skilled in the art have not been described in detail in order to highlight the spirit of the invention.

[0076] In an optional embodiment, textual description information is input into a target diffusion model, and a target image is generated by performing reverse denoising inference through the target diffusion model; an acceleration mechanism is enabled during the reverse denoising inference process through the target diffusion model.

[0077] In one specific embodiment, the text description information can be text used to guide the target diffusion model in generating the target image; the target diffusion model can be determined according to actual application requirements, specifically, the target diffusion model can be StableDiffusion. Enabling an acceleration mechanism during the reverse denoising inference process using the target diffusion model can maximize the generation quality while improving the inference speed of the diffusion model. Optionally, other multimodal information such as images can also be input into the target diffusion model, and the target diffusion model can perform reverse denoising inference to generate the actual required content.

[0078] Figure 1 This is a flowchart illustrating an acceleration mechanism enabled during reverse denoising inference using the target diffusion model, as provided in this embodiment of the invention. This specification provides method steps as shown in the embodiments or flowcharts, but based on conventional or non-inventive methods, more or fewer steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only possible execution order. In actual system or server product execution, the methods shown in the embodiments or drawings can be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment). Specifically, as shown... Figure 1 As shown, the above method may include:

[0079] S101: Based on the non-uniform time step sequence, determine the target number of key time steps;

[0080] In a specific embodiment, the non-uniform time step sequence is obtained by solving the time optimization objective function. Optionally, the non-uniform time step sequence can be a sequence of sampling time-series nodes for inverse denoising of the diffusion model. Specifically, the non-uniform time step sequence includes multiple sampling time-series nodes. The time optimization objective function is constructed based on the truncation error generated by the ODE (Ordinary Differential Equation) solver of the target diffusion model. Optionally, the time step optimization objective function aims to minimize the cumulative truncation error associated with each time step. Optionally, the time step optimization objective function is solved to generate the non-uniform time step sequence. The key time step is the time step where the local truncation error is greater than or equal to a preset threshold. Optionally, the preset threshold can be set according to actual application requirements.

[0081] In an optional embodiment, the above time-step optimization objective function is constructed using the following method:

[0082] Based on the truncation error generated by the ODE solver of the target diffusion model, the initial time step optimization objective function is constructed as shown in Equation (1):

[0083] (1);

[0084] The constraints for the initial time step optimization objective function are: ;

[0085] in, Indicates that ODE is at time step The true solution, This indicates that the ODE solver is at time step The approximate solution, the truncation error generated by the ODE solver is determined based on the approximate solution and the true solution; Indicates the total number of sampling steps; Indicates the first Each time step Indicates the first Each time step 0 represents the initial time step, and 0 represents the final time step;

[0086] Based on the initial time step optimization objective function, an upper bound inequality for the error between the true solution and the approximate solution is constructed, as shown in equation (2):

[0087] (2);

[0088] in, This represents the approximate solution of the ODE solver at the final time step. This represents the true solution of ODE at the final time step; This represents the noise standard deviation at the final time step. The noise standard deviation represents the initial time step; This represents the half-log signal-to-noise ratio at the final time step. This represents the semi-log signal-to-noise ratio at the initial time step; This represents a constant related to the order of the ODE solver; Indicates time step Local truncation error; This represents the multi-step weighting coefficients of the ODE solver. This represents the global index of the current time step. Indicates the order of the ODE solver. This represents the history offset index of the ODE solver;

[0089] Based on the error upper bound inequality, the time step optimization objective function is determined as shown in equation (3):

[0090] (3);

[0091] The constraints for the time-step optimization objective function are: ;

[0092] in, This represents the first time step in a non-uniform time step sequence. The corresponding semi-log signal-to-noise ratio, In a non-uniform time step sequence, the th time step The corresponding semi-log signal-to-noise ratio, In a non-uniform time step sequence, the th time step The corresponding semi-log signal-to-noise ratio, In a non-uniform time step sequence, the th time step The corresponding semi-logarithmic signal-to-noise ratio.

[0093] In a specific embodiment, by constructing a diffusion model noise scheduling mechanism, the forward calculation and reverse solution of the marginal mean coefficient, noise standard deviation, and semi-logarithmic signal-to-noise ratio are achieved. The constructed diffusion model noise scheduling mechanism can include three noise scheduling types: discrete, linear, and cosine, and is compatible with the noise scheduling configurations of various pre-trained diffusion models. Optionally, the marginal mean coefficient is the signal attenuation coefficient during diffusion, the noise standard deviation is the noise intensity coefficient during diffusion, and the semi-logarithmic signal-to-noise ratio is an intermediate variable for time step optimization. Specifically, the semi-logarithmic signal-to-noise ratio is shown in the following formula:

[0094] ,in, Indicates time step The half-log signal-to-noise ratio, Represents the marginal mean coefficient. It represents the standard deviation.

[0095] In one specific embodiment, a cosine noise schedule for Stable Diffusion can be constructed, and then, based on this cosine noise schedule, the noise standard deviation of the initial time step, the noise standard deviation of the final time step, the semi-logarithmic signal-to-noise ratio of the initial time step, and the semi-logarithmic signal-to-noise ratio of the final time step can be calculated. Optionally, the constructed upper bound inequality of error introduces a noise schedule for the diffusion model.

[0096] In a specific embodiment, the ODE solver, also known as the target sampler, can be the sampler corresponding to the diffusion model. Optionally, the initial time-step optimization objective function can aim to minimize the cumulative squared error between the true solution and the approximate solution across all sampling steps. Specifically, the optimization objective can be to use the true solution as the baseline truth value, making the approximate solution infinitely close to the true solution. Optionally, the total number of sampling steps... This represents the total number of native sampling steps, which can be set according to actual application requirements; the initial time step can be the maximum time value of the diffusion process, at which point the noise is strongest; the final time step can be the minimum time value of the diffusion process, at which point denoising is complete.

[0097] To simplify the optimization process, an upper bound inequality between the approximate and true solutions can be determined based on the initial time-step objective function, thus simplifying the optimization objective to minimizing the time-step-dependent error term. Optionally, the final true solution of the sampler is the denoised, clean image. This can be the total error of the final generated result; This is an initial error term that is independent of time step scheduling and can be ignored during optimization. The cumulative truncation error term related to time step scheduling can be used to solve for non-uniform time step sequences, where, Positively correlated with the time step interval; optionally, a constant related to the ODE solver order. The multi-step weighting coefficients of the ODE solver can be determined based on practical applications. This can be determined based on the actual sampler type; optionally, the global sequence number of the current time step. This can be the index of the current time step within the total number of sampling steps; or the order of the ODE solver. Based on the actual solver, when the ODE solver is PLMS (Pseudo Linear Multistep Sampler), PLMS is a 3rd-order solver. ODE solver history offset index It can be used to iterate through all historical time steps used in the current time step; optional, The first variable to be optimized in the objective function can be used to optimize the time step. The last variable to be optimized in the objective function can be optimized for each time step. These can be variables representing the signal-to-noise ratio at the next time step in the constraints of the objective function for time step optimization. It can be a variable representing the signal-to-noise ratio at the current time step in the constraints of the objective function for time step optimization.

[0098] Based on the error upper bound inequality, the time step optimization objective function can be determined. Specifically, the time step optimization objective function aims to minimize the cumulative truncation error term related to the time step, thereby simplifying the complex global error optimization problem into minimizing the cumulative truncation error term related to the time step. Then, based on the time step optimization objective function, the non-uniform time step sequence can be solved, thus providing an error quantification basis for subsequent key time step selection.

[0099] In a specific embodiment, the time step optimization objective function is solved using a trust region method with linear constraints to generate a non-uniform time series. Specifically, the semi-log signal-to-noise ratio (SNR) is adjusted at each intermediate time step under the condition that the semi-log SNR increases monotonically during the reverse denoising process. The semi-log SNR sequence that minimizes the cumulative truncation error term is then obtained. Based on the mapping relationship between the semi-log SNR and the time step, the semi-log SNR sequence is transformed into a non-uniform time step sequence.

[0100] In an optional embodiment, determining the target number of key time steps based on a non-uniform time step sequence may include:

[0101] Based on the local truncation error of each non-uniform time step, time step screening is performed on the non-uniform time step sequence to determine the set of candidate key time steps, as shown in Equation (4):

[0102] (4);

[0103] in, Represents the set of candidate key time steps; Indicates non-uniform time steps Local truncation error; This represents the maximum value of the local truncation error for all non-uniform time steps; This represents the filtering threshold coefficient; Represents the total number of non-uniform time intervals;

[0104] From the set of candidate key time steps, retain the target number of key time steps; where the target number of key time steps is as shown in equation (5):

[0105] (5);

[0106] in, Indicates the number of targets at key time steps. This represents the global retention ratio. This indicates the total number of sampling steps.

[0107] In one specific embodiment, the local truncation error of each time step is determined based on the cumulative truncation error term described above. The candidate key time step set can be a set of key time steps initially selected from the non-uniform time step sequence based on the local truncation error of each time step. A screening threshold coefficient is then used. It can be set according to actual application needs; it is optional. The value range can be [0.1, 0.5]. Specifically, , Used to control the number of candidate key time steps in order to balance the number of key time steps with the quality of generation; A benchmark value can be used to filter key time steps based on a threshold of local truncation error. Specifically, based on the cumulative truncation error term of the aforementioned error upper bound inequality, the local truncation error of each non-uniform time step in the non-uniform time step sequence is determined, providing an error quantification basis for subsequent key time step screening. The local truncation errors of each non-uniform time step are sorted in descending order to obtain the first truncation error descending order result. The first local truncation error of the first truncation error descending order result is determined as the maximum value of the local truncation errors of all non-uniform time steps. An adaptive screening threshold is obtained by multiplying the screening threshold coefficient by the maximum value of the local truncation errors of all non-uniform time steps. Optionally, the aforementioned preset threshold can be the adaptive screening threshold. Based on the local truncation error of each non-uniform time step and the adaptive screening threshold, time step screening is performed on the non-uniform time step sequence to obtain a set of candidate key time steps, thereby realizing dynamic screening of key time steps. Optionally, by screening non-uniform time steps with local truncation errors greater than the adaptive screening threshold as candidate key time steps, time steps that have a significant impact on the denoising result can be determined.

[0108] In one specific embodiment, by Determine the total number of critical time steps to be retained. Global retention ratio coefficient. It can be configured according to hardware performance and quality requirements; it is optional. , specifically , Used to determine the total number of key time steps to be retained, specifically in At that time, one can obtain Specifically, 12 key time steps are retained from the candidate key time step set. Then, the local truncation errors of each candidate key time step set are sorted in descending order to obtain a second truncation error descending order result; the top target number of candidate key time steps in the second truncation error descending order result are determined as the target number of key time steps. Optionally, the number of candidate key time steps in the candidate key time step set is less than or equal to the target number. If so, the set of candidate key time steps is directly determined as the set of key time steps.

[0109] In one specific embodiment, time steps in a non-uniform time step sequence with relatively small local stage errors, excluding critical time steps, can be considered non-critical time steps.

[0110] In the above embodiments, key time steps and non-key time steps are identified. By performing full computation at key time steps and reusing cached features to skip computation at non-key time steps, synergistic efficiency can be achieved between the two, and the number of times the UNet full forward pass of the target diffusion model is executed is reduced, thereby improving inference speed.

[0111] S102: Based on the preset early-stage mapping protection parameters, determine the set of main key time steps from the target number of key time steps;

[0112] In an optional embodiment, the above-mentioned set of main key time steps is determined from the target number of key time steps based on preset early-stage mapping protection parameters, as shown in equation (7):

[0113] (7);

[0114] in, This represents the set of key time steps. This indicates the number of key time steps in the target sequence. A key time step; This indicates the preset parameters for protecting the initial layout. This represents the total number of critical time steps.

[0115] In a specific embodiment, the preset early-stage composition protection parameter can be set according to actual application requirements. Optionally, the value range of the preset early-stage composition protection parameter can be [0,1]. Specifically, the preset early-stage composition protection parameter is 0.3. The preset early-stage composition protection parameter can be used to define the length ratio of the early-stage composition protection phase; in At that time, the first 30% × 12 ≈ 3 critical time steps are the main critical time steps. The set of main critical time steps can be the set of time steps in the early stage of diffusion that require mandatory full execution of forward computation.

[0116] In the above embodiments, by forcing the encoder to perform full forward computation at all major key time steps within the determined set of major key time steps, and prohibiting any skipping steps and cache reuse, early graph protection is achieved to ensure that the graph information in the early stages of diffusion is completely extracted, thereby avoiding graph collapse and loss under high speedup ratios that lead to quality degradation.

[0117] S103: The encoder and decoder of the target diffusion model perform full forward computation based on the number of key time steps and the set of main key time steps of the target, and combine with the asymmetric caching strategy to generate corresponding target features, and update the cached features to target features.

[0118] In a specific embodiment, the diffusion model infrastructure may include a UNet (U-Net, U-shaped network) module. Optionally, the UNet includes an encoder and a decoder. The encoder is used to extract the semantic structure and overall composition information of the input, and the decoder is used to generate image details and texture information based on high-dimensional features. Optionally, the encoder includes a downsampling module and an intermediate module. Specifically, the encoder can be used to extract low-level to high-level semantic features and overall composition features from the input noisy image. Optionally, the decoder includes an upsampling module. Specifically, the decoder can be used to generate image details and texture features based on the high-dimensional features output by the encoder. By using residual connections between the encoder's same-scale module and the corresponding module of the decoder, the same-scale features of the encoder and decoder are concatenated, thereby transmitting low-level edge and contour features to avoid loss of details during the upsampling process.

[0119] In a specific embodiment, the asymmetric caching strategy may include a conservative encoder reuse strategy and an aggressive decoder update strategy. Optionally, the conservative encoder reuse strategy may involve performing a full forward computation of the encoder and updating the encoder cache when the current non-uniform time step is a critical time step and the current critical time step meets a preset condition; critical time steps that do not meet the preset condition reuse cached features. Optionally, the preset condition can be set according to the actual application; specifically, the preset condition may be that the sequence number of the critical time step in the global critical time step is an integer multiple of a preset skip step parameter. The encoder is used for structural graph information extraction and adopts a conservative strategy, performing a full forward computation at fixed intervals of critical time steps to balance structural fidelity and acceleration. Optionally, the aggressive decoder update strategy may involve performing a full forward computation of the decoder at critical time steps and updating the decoder cache; the decoder is used for detailed texture generation, with high feature redundancy between adjacent time steps to reduce the impact on generation quality and minimize computation.

[0120] In a specific embodiment, the target feature may include the high-level semantic structure feature corresponding to the encoder and the intermediate high-level texture feature corresponding to the decoder. Specifically, the target feature is related to the module currently performing full forward computation. When performing full forward computation through the encoder, the target feature may be the high-level semantic structure feature corresponding to the encoder. When performing full forward computation through the decoder, the target feature may be the intermediate high-level texture feature corresponding to the decoder.

[0121] In the above embodiments, an acceleration framework that deeply integrates critical time step optimization and deep cache reuse is used to perform full computation at critical time steps and reuse cached features to skip computation at non-critical time steps, thereby achieving synergistic efficiency between the two and reducing the number of times UNet performs full forward pass, thus improving inference speed.

[0122] In an optional embodiment, the encoder and decoder of the target diffusion model, based on the target number of key time steps and the set of major key time steps, and combined with an asymmetric caching strategy, perform full forward computation to generate corresponding target features, and updating the cached features to the target features may include:

[0123] The encoder performs full forward computation at each of the major key time steps in the set of major key time steps to generate high-level semantic structure features for the corresponding key time steps, and updates the cached features to the high-level semantic structure features.

[0124] After performing full forward computation at each major key time step, the encoder performs full forward computation at non-major key time steps that meet preset conditions in the target number of key time steps, generates high-level semantic structure features for the corresponding key time steps, and updates the cached features to high-level semantic structure features.

[0125] The decoder performs full forward computation at each of the target number of key time steps to generate intermediate high-level texture features for the corresponding key time steps, and updates the cached features to the intermediate high-level texture features.

[0126] Reusing cached target features at non-critical time steps via encoder and decoder, including:

[0127] The encoder reuses cached high-level semantic structure features at non-critical time steps;

[0128] The decoder reuses cached intermediate high-level texture features at non-critical time steps.

[0129] In a specific embodiment, executable caching operations can be constructed, including: high-level feature caching operations, low-level feature update operations, and jump-connection feature synchronization operations. Optionally, the high-level feature caching operations include caching the high-level semantic features output by the encoder and the intermediate high-level texture features output by the decoder for subsequent reuse. The low-level feature update operations include updating the low-level pixel features output by the encoder and the low-level contour features of the jump-connection branches of the decoder at each step to avoid loss of basic contour information. The jump-connection feature synchronization operations include caching the features of the jump-connection branches of the encoder to the decoder at the same scale, updating and reusing them synchronously with the high-level semantic features of the encoder to ensure temporal alignment between the encoder structural features and the decoder detailed features.

[0130] In one specific embodiment, by performing full forward computation at each of the major key time steps in the set of major key time steps by the encoder, high-level semantic structure features for the corresponding key time steps are generated, and the cached features are updated to high-level semantic structure features. This ensures the complete extraction of early graphing information and improves both the accuracy of early graphing and inference speed. After performing full forward computation at each major key time step, the encoder performs full forward computation at non-major key time steps that meet preset conditions within the target number of key time steps, generating high-level semantic structure features for the corresponding key time steps, and updating the cached features to high-level semantic structure features. Hierarchical semantic structure features can further improve the speedup ratio. By performing full forward computation at each key time step in the target number of key time steps by the decoder, intermediate high-level texture features for the corresponding key time steps are generated, and the cached features are updated to intermediate high-level texture features, which can preserve rich texture details. The encoder reuses the cached high-level semantic structure features in non-key time steps. The decoder reuses the cached intermediate high-level texture features in non-key time steps. Thus, while improving inference speedup, it can ensure the integrity of the subject structure, accurate semantic alignment, and preservation of rich texture details of the generated image, thereby taking into account both the global structure and local details of the image.

[0131] In one specific embodiment, the encoder updates low-level pixel features at each non-uniform time step, and the decoder updates low-level contour features of jump-connected branches at each non-uniform time step; through jump-connected feature synchronization operation, while caching the updated high-level semantic structure features, the features of the jump-connected branches from the encoder to the decoder at the same scale are cached and updated, and while reusing the high-level semantic structure features, the features of the jump-connected branches from the encoder to the decoder at the same scale are reused.

[0132] In an optional embodiment, performing full forward computation through the encoder to generate high-level semantic structure features corresponding to key time steps, and updating the cached features to high-level semantic structure features may include:

[0133] Iterate through each non-uniform time step in the non-uniform time step sequence;

[0134] When traversing to any non-uniform time step, based on the global time step mapping table, determine whether the current non-uniform time step is a critical time step; the global time step mapping table is based on the non-uniform time step sequence and the target number of critical time steps, and includes the value of each non-uniform time step and the value representing whether the corresponding non-uniform time step is a critical time step.

[0135] When the current non-uniform time step is a critical time step, the sequence number of the current critical time step is obtained based on the global critical step mapping table; the global critical step mapping table is generated according to the target number of critical time steps.

[0136] Based on the current critical time step number, determine whether the current critical time step belongs to the set of major critical time steps;

[0137] When the current critical time step belongs to the set of major critical time steps, the encoder performs full forward computation at the current critical time step to generate high-level semantic structure features for the corresponding critical time step, and updates the cached features to high-level semantic structure features.

[0138] If the current critical time step does not belong to the set of major critical time steps, determine whether the sequence number of the current critical time step is an integer multiple of the preset skip step parameter;

[0139] When the sequence number of the current key time step is an integer multiple of the preset skip step parameter, the encoder performs full forward computation at the current key time step to generate the high-level semantic structure features of the corresponding key time step, and updates the cached features to the high-level semantic structure features.

[0140] In a specific embodiment, the global time step mapping table may include the value of each non-uniform time step and its corresponding discrete index, as well as the value representing whether each non-uniform time step is a critical time step; optionally, the global time step mapping table may represent the discrete index corresponding to each non-uniform time step and whether the non-uniform time step corresponding to the discrete index is a critical time step; the global critical step mapping table may represent the sequence number of each critical time step among all critical time steps; the preset skip step parameter may be a parameter used to control the encoder skip step, used to balance the acceleration effect and the generation quality, and at the same time realize the asymmetric scheduling of the encoder and decoder. The preset skip step parameter can be set according to the actual application requirements. Specifically, the value range of the preset skip step parameter may be [1,5]. Specifically, based on the global time step mapping table, it is determined whether the current non-uniform time step is a critical time step. If the current non-uniform time step is a critical time step, the index of the current critical time step is obtained based on the global critical time step mapping table. Then, based on the index of the current critical time step, it is determined whether the current critical time step belongs to the main critical time step set. If the current critical time step belongs to the main critical time step set, the encoder performs full forward computation at the current critical time step to generate the high-level semantic structure features corresponding to the critical time step, and updates the cached features to the high-level semantic structure features. If the current critical time step does not belong to the main critical time step set... In the following cases, determine whether the current key time step number is an integer multiple of the preset skip step parameter; if the current key time step number is an integer multiple of the preset skip step parameter, perform full forward computation at the current key time step through the encoder to generate the corresponding high-level semantic structure features, and update the cached features to the high-level semantic structure features; if the current key time step number is not an integer multiple of the preset skip step parameter, reuse the cached high-level semantic structure features at the current key time step through the encoder; if the current non-uniform time step is not a key time step, reuse the cached high-level semantic structure features at the current non-key time step through the encoder.

[0141] In one specific embodiment, an encoder execution decision table is constructed based on the sequence number of each key time step. This table characterizes whether a full forward computation of the encoder is performed at each key time step; 1 indicates full forward computation, and 0 indicates cache reuse. Specifically, based on the sequence number of the current key time step, if it belongs to the set of major key time steps, it is marked as 1. If it does not belong to the set of major key time steps, but its sequence number is an integer multiple of a preset skip parameter, it is marked as 1. Furthermore, if it does not belong to the set of major key time steps and its sequence number is not an integer multiple of the preset skip parameter, it is marked as 0. This process yields the encoder execution decision table. Specifically, when the current non-uniform time step is a key time step, its sequence number is obtained based on the global key step mapping table. Then, the encoder performs the corresponding operation based on the sequence number of the current key time step and the encoder execution decision table. Optionally, a cache reuse index table can be constructed. The cache reuse index table can record the cache traceability index corresponding to non-critical time steps and encoder skip key steps, which can be used to locate the features to be reused in non-critical time steps and encoder skip key steps.

[0142] In one specific embodiment, by coordinating the target number of key time steps, preset early-stage graph protection parameters, and preset skip step parameters, it can cover all scenario requirements from maximizing quality to maximizing speed, without requiring any fine-tuning or retraining of the model. It is plug-and-play and can be seamlessly compatible with all existing mainstream fast samplers.

[0143] In an optional embodiment, performing full forward computation through the decoder to generate intermediate high-level texture features corresponding to key time steps, and updating the cached features to intermediate high-level texture features may include:

[0144] When the current non-uniform time step is the critical time step, the decoder performs full forward computation at the current critical time step to generate intermediate high-level texture features for the corresponding critical time step, and updates the cached features to the intermediate high-level texture features.

[0145] In one specific embodiment, based on the global time step mapping table, it is determined whether the current non-uniform time step is a critical time step; if the current non-uniform time step is a critical time step, the decoder performs full forward computation at the current critical time step to generate intermediate high-level texture features corresponding to the critical time step, and updates the cached features to the intermediate high-level texture features; if the current non-uniform time step is not a critical time step, the decoder reuses the cached intermediate high-level texture features at the current non-critical time step.

[0146] In the above embodiments, the table lookup operation can reduce additional overhead and achieve fine control over speedup and generation quality.

[0147] In a specific embodiment, the effective execution steps of the encoder and decoder can be further defined, and the effective execution steps of the encoder and decoder can be an upper limit of the number of times the encoder and decoder are executed; specifically, the effective execution steps of the encoder and decoder are shown in the following formula:

[0148] ;

[0149] ;

[0150] in, Indicates the number of valid execution steps of the encoder. This indicates the preset jump parameters. This indicates the number of valid execution steps of the decoder.

[0151] Optionally, the fewer effective execution steps the encoder has, the higher the step skipping frequency and the less computation required; the more effective execution steps the decoder has, the higher the full computation frequency and the stronger the detail protection, thereby achieving differentiated scheduling of the encoder and decoder.

[0152] S104: Reuse cached target features at non-critical time steps via encoder and decoder.

[0153] In an optional embodiment, the above method may further include:

[0154] The cache reuse interval corresponding to each critical time step is determined as shown in Equation (6):

[0155] (6);

[0156] in, Indicates the first The cache reuse interval corresponding to each key time step; Indicates the cache interval adjustment factor; Indicates the first Key time steps and the first The time interval between two adjacent key time steps; Indicates the first The noise standard deviation of each key time step;

[0157] For each critical time step, the target features cached at the current critical time step are reused in the non-critical time steps corresponding to the cache reuse interval of the current critical time step by the encoder and decoder;

[0158] If there are non-critical time steps after the cache reuse interval of the current critical time step, the encoder and decoder perform full forward computation at the first non-critical time step after the cache reuse interval to generate the corresponding target features, and update the cached features to the target features.

[0159] The encoder and decoder reuse the cached target features in non-critical time steps after the first non-critical time step until the next critical time step, when the encoder and decoder perform full forward computation to generate the corresponding target features and update the cached features to the target features.

[0160] In one specific embodiment, the cache reuse interval for each key time step can be determined based on the interval between key time steps and the noise intensity. Optionally, the cache reuse interval can be an upper limit for the number of safe reuses to avoid errors exceeding the acceptable range. Optionally, the cache reuse interval for the current key time step can characterize the number of non-key time steps for which the feature corresponding to the current key time step can be reused. Cache interval adjustment factor It can be configured according to actual application needs, specifically... You can go to version 1.0. Specifically, in the early stage of diffusion, the noise intensity is high and the features change rapidly, resulting in a small buffer reuse interval; in the later stage of diffusion, the noise intensity is low and the feature redundancy is high, resulting in a large buffer reuse interval.

[0161] In one specific embodiment, if the interval between the current critical time step and the next critical time step is less than or equal to the cache reuse interval, the encoder and decoder reuse the target features cached in the current critical time step in the non-critical time step corresponding to the cache reuse interval of the current critical time step. Optionally, if the interval between the current critical time step and the next critical time step is greater than the cache reuse interval, the encoder and decoder reuse the target features cached in the current critical time step in the non-critical time step corresponding to the cache reuse interval of the current critical time step. The encoder and decoder perform full forward computation in the first non-critical time step after the cache reuse interval to generate the corresponding target features, and update the cached features to the target features. ; and reuse the target features of the cached cache in the cache reuse interval corresponding to the first non-critical time step through the encoder and decoder; determine whether there are still non-critical time steps after the cache reuse interval of the first non-critical time step. If there are still non-critical time steps after the cache reuse interval of the first non-critical time step, repeat the above steps of performing full forward computation in the first non-critical time step after the cache reuse interval through the encoder and decoder until the above steps of determining whether there are still non-critical time steps after the cache reuse interval of the first non-critical time step are repeated, until in the next critical time step, perform full forward computation through the encoder and decoder to generate the corresponding target features respectively, and update the cached features to the target features.

[0162] In one specific implementation, after the diffusion model inference is completed, the model's original state is restored, and GPU memory resources are released. Specifically, after the diffusion model inference is completed, all accelerated modifications can be reversed, and all temporarily occupied resources can be cleaned up to return the model to its original, unaccelerated state, thus avoiding GPU memory leaks.

[0163] In a specific embodiment, the acceleration effect can be quantified by determining the theoretical speedup factor of the above diffusion model inference acceleration method over native inference, as shown in the following formula:

[0164] ;

[0165] in, Indicates the theoretical acceleration factor; Indicates the total number of original sampling steps; This represents the computational cost and time consumption of a single-step full forward computation of UNet at a critical time step; This indicates the computational cost and time consumption for reusing the cache in non-critical time steps.

[0166] Optionally, the change in generated quality can be quantified using a quality loss assessment function, as shown in the following equation:

[0167] ;

[0168] in, This represents the overall quality loss value, which is optional. The smaller the overall quality loss value, the closer the generated quality is to the original inference. It represents the feature distribution distance between the generated image and the real image, and can be used to measure the fidelity of the generation; This indicates the similarity between the generated image and the input text, and can be used to measure text alignment capabilities. and These represent the first and second weighting coefficients, respectively, which can be used to balance fidelity and text alignment capabilities.

[0169] As can be seen from the technical solutions provided in the embodiments of this specification above, this specification inputs textual description information into the target diffusion model, performs reverse denoising inference through the target diffusion model, and generates a target image. Enabling an acceleration mechanism during the reverse denoising inference process through the target diffusion model can maximize the generation quality while improving the inference speed of the diffusion model. Specifically, enabling the acceleration mechanism during the reverse denoising inference process through the target diffusion model includes: determining a target number of key time steps based on a non-uniform time step sequence; the non-uniform time step sequence is obtained by solving a time optimization objective function, which is constructed based on the truncation error generated by the ODE solver of the target diffusion model; and key time steps are those where the local truncation error is greater than or equal to a preset threshold. Based on preset early-stage composition protection parameters, a set of main key time steps is determined from the target number of key time steps. This avoids the loss of composition information in the early stages of diffusion, which can lead to a decrease in generation quality, thereby significantly improving the generation quality in high-speed-ratio scenarios. The system achieves robustness and semantic alignment accuracy. It utilizes the encoder and decoder of the target diffusion model to perform full forward computation based on the number of key time steps and the set of major key time steps, combined with an asymmetric caching strategy. This generates corresponding target features and updates the cached features to the target features. Furthermore, it accelerates the system by reusing cached target features at non-key time steps, leveraging a deep fusion of key time step optimization and deep cache reuse. The asymmetric caching strategy reduces the number of executions, improves acceleration efficiency, and balances structural fidelity with detail generation, thus achieving a balance between acceleration and generation quality. This significantly reduces the inference latency of the diffusion model while maximizing generation quality. Consequently, it achieves a substantial reduction in inference latency and significant compression of GPU memory usage without model retraining, loss of generation quality, or loss of structural integrity, semantic alignment, and detail richness of the generated image. It is also compatible with various open-source pre-trained models.

[0170] This invention also provides a diffusion model inference acceleration device, correspondingly, Figure 2This is a schematic diagram of the structure of a diffusion model inference acceleration device provided in an embodiment of the present invention; as shown below. Figure 2 As shown, the above-mentioned device includes:

[0171] The model inference module 210 is used to input text description information into the target diffusion model, and perform reverse denoising inference through the target diffusion model to generate a target image.

[0172] The model inference module 210 includes an inference acceleration module 220, which is used to enable an acceleration mechanism during the reverse denoising inference process through the target diffusion model.

[0173] The inference acceleration module 220 includes:

[0174] The time step optimization unit is used to determine the target number of key time steps based on a non-uniform time step sequence. The non-uniform time step sequence is obtained by solving the time optimization objective function, which is constructed based on the truncation error generated by the ODE solver of the target diffusion model. The key time steps are the time steps where the local truncation error is greater than or equal to a preset threshold.

[0175] The cache integration unit is used to determine the set of main key time steps from the target number of key time steps based on preset early-stage mapping protection parameters; to perform full forward computation by the encoder and decoder of the target diffusion model according to the target number of key time steps and the set of main key time steps, and in combination with an asymmetric caching strategy, to generate corresponding target features respectively, and to update the cached features with the target features; and to reuse the cached target features in non-key time steps by the encoder and the decoder.

[0176] This invention also provides an electronic device, which includes a processor and a memory. The memory stores at least one instruction, at least one program, a code set, or an instruction set. The at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the diffusion model inference acceleration method as described in any of the method embodiments.

[0177] Embodiments of the present invention also provide a computer storage medium, which may be disposed in a server to store at least one instruction, at least one program, code set, or instruction set for implementing the method embodiments. The at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the diffusion model inference acceleration method as described in any of the method embodiments.

[0178] Optionally, in embodiments of the present invention, the storage medium may be located at at least one of a plurality of network servers in a computer network. Optionally, in embodiments of the present invention, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0179] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0180] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more flowcharts and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0181] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more flowcharts and / or boxes Figure 1 The function specified in one or more boxes.

[0182] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more flowcharts and / or boxes Figure 1 Figure 1 The steps of the function specified in one or more boxes.

[0183] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0184] Finally, it should be noted that the embodiments of the present invention have been described above in conjunction with the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. A method for accelerating inference in a diffusion model, characterized in that, The method includes: The text description information is input into the target diffusion model, and the target image is generated by reverse denoising inference through the target diffusion model; The acceleration mechanism enabled during the reverse denoising inference process using the target diffusion model includes: Based on a non-uniform time step sequence, a number of key time steps are determined for the target. The non-uniform time step sequence is obtained by solving a time step optimization objective function, which is constructed based on the truncation error generated by the ODE solver of the target diffusion model. The key time steps are those where the local truncation error is greater than or equal to a preset threshold. The time step optimization objective function is constructed using the following method: Based on the truncation error generated by the ODE solver of the target diffusion model, the initial time step optimization objective function is constructed as shown in equation (1): （1）； The constraints for the initial time step optimization objective function are as follows: ; in, Indicates that ODE is at time step The true solution, This indicates that the ODE solver is at time step The approximate solution is obtained, and the truncation error generated by the ODE solver is determined based on the approximate solution and the true solution. Indicates the total number of sampling steps; Indicates the first Each time step Indicates the first Each time step 0 represents the initial time step, and 0 represents the final time step; Based on the initial time step optimization objective function, the error upper bound inequality between the true solution and the approximate solution is constructed, as shown in equation (2): （2）； in, This represents the approximate solution of the ODE solver at the final time step. This represents the true solution of ODE at the final time step; This represents the noise standard deviation at the final time step. The noise standard deviation represents the initial time step; This represents the half-log signal-to-noise ratio at the final time step. This represents the semi-log signal-to-noise ratio at the initial time step; This represents a constant related to the order of the ODE solver; Indicates time step Local truncation error; This represents the multi-step weighting coefficients of the ODE solver. This represents the global index of the current time step. Indicates the order of the ODE solver. This represents the history offset index of the ODE solver; Based on the error upper bound inequality, the time step optimization objective function is determined as shown in equation (3): （3）； The constraints of the time-step optimization objective function are as follows: ; in, This represents the first time step in a non-uniform time step sequence. The corresponding semi-log signal-to-noise ratio, In a non-uniform time step sequence, the th time step The corresponding semi-log signal-to-noise ratio, In a non-uniform time step sequence, the th time step The corresponding semi-log signal-to-noise ratio, In a non-uniform time step sequence, the th time step The corresponding semi-log signal-to-noise ratio; Based on preset early-stage mapping protection parameters, a set of main key time steps is determined from the target number of key time steps; The encoder and decoder of the target diffusion model generate corresponding target features by performing full forward computation based on the target number of key time steps and the set of main key time steps, combined with an asymmetric caching strategy, and update the cached features to the target features. And the target features cached are reused at non-critical time steps by the encoder and the decoder.

2. The diffusion model inference acceleration method according to claim 1, characterized in that, The determination of the target number of key time steps based on a non-uniform time step sequence includes: Based on the local truncation error of each non-uniform time step, the non-uniform time step sequence is filtered to determine the set of candidate key time steps, as shown in equation (4): （4）； in, Represents the set of candidate key time steps; Indicates non-uniform time steps Local truncation error; This represents the maximum value of the local truncation error for all non-uniform time steps; This represents the filtering threshold coefficient; Represents the total number of non-uniform time intervals; From the set of candidate key time steps, retain the target number of key time steps; wherein, the target number of key time steps is as shown in equation (5): （5）； in, Indicates the number of targets at key time steps. This represents the global retention ratio. This indicates the total number of sampling steps.

3. The diffusion model inference acceleration method according to claim 1, characterized in that, The target features include the high-level semantic structure features corresponding to the encoder and the intermediate high-level texture features corresponding to the decoder; The encoder and decoder of the target diffusion model generate corresponding target features based on the target number of key time steps and the set of main key time steps, and perform full forward computation in conjunction with an asymmetric caching strategy, and update the cached features with the target features, including: The encoder performs full forward computation at each of the major key time steps in the set of major key time steps to generate high-level semantic structure features for the corresponding key time steps, and updates the cached features with the high-level semantic structure features. After performing full forward computation at each major key time step, the encoder performs full forward computation at non-major key time steps that meet preset conditions in the target number of key time steps to generate high-level semantic structure features for the corresponding key time steps, and updates the cached features to the high-level semantic structure features. The decoder performs full forward computation at each of the target number of key time steps to generate intermediate high-level texture features for the corresponding key time steps, and updates the cached features with the intermediate high-level texture features. The reuse of cached target features by the encoder and the decoder at non-critical time steps includes: The encoder reuses the high-level semantic structure features of the cache at the non-critical time steps; The decoder reuses the intermediate high-level texture features of the cache at the non-critical time step.

4. The diffusion model inference acceleration method according to claim 3, characterized in that, The encoder performs a full forward computation to generate high-level semantic structure features corresponding to key time steps, and updates the cached features with these high-level semantic structure features, including: Iterate through each non-uniform time step in the non-uniform time step sequence; When traversing to any non-uniform time step, based on the global time step mapping table, it is determined whether the current non-uniform time step is a critical time step; the global time step mapping table is determined according to the non-uniform time step sequence and the target number of critical time steps, and the global time step mapping table includes the value of each non-uniform time step and the value representing whether the corresponding non-uniform time step is a critical time step. When the current non-uniform time step is a critical time step, the sequence number of the current critical time step is obtained based on the global critical step mapping table; the global critical step mapping table is generated according to the target number of critical time steps. Based on the sequence number of the current critical time step, determine whether the current critical time step belongs to the set of main critical time steps; When the current key time step belongs to the set of main key time steps, the encoder performs full forward computation at the current key time step to generate high-level semantic structure features for the corresponding key time step, and updates the cached features to the high-level semantic structure features. If the current critical time step does not belong to the set of main critical time steps, determine whether the sequence number of the current critical time step is an integer multiple of the preset skip step parameter; When the sequence number of the current key time step is an integer multiple of the preset skip step parameter, the encoder performs full forward computation at the current key time step to generate high-level semantic structure features corresponding to the key time step, and updates the cached features to the high-level semantic structure features.

5. The diffusion model inference acceleration method according to claim 4, characterized in that, The decoder performs a full forward computation to generate intermediate high-level texture features corresponding to key time steps, and updates the cached features with these intermediate high-level texture features, including: When the current non-uniform time step is a critical time step, the decoder performs full forward computation at the current critical time step to generate intermediate high-level texture features corresponding to the critical time step, and updates the cached features to the intermediate high-level texture features.

6. The diffusion model inference acceleration method according to claim 1, characterized in that, The method further includes: The cache reuse interval corresponding to each critical time step is determined as shown in Equation (6): （6）； in, Indicates the first The cache reuse interval corresponding to each key time step; Indicates the cache interval adjustment factor; Indicates the first Key time steps and the first The time interval between two adjacent key time steps; Indicates the first The noise standard deviation of each key time step; For each critical time step, the encoder and decoder reuse the target features cached in the current critical time step in the non-critical time step corresponding to the cache reuse interval of the current critical time step; If there are non-critical time steps after the cache reuse interval of the current critical time step, the encoder and the decoder perform full forward computation at the first non-critical time step after the cache reuse interval to generate corresponding target features, and update the cached features to the target features. The encoder and decoder reuse the cached target features in non-critical time steps after the first non-critical time step until the next critical time step. Then, the encoder and decoder perform full forward computation to generate the corresponding target features and update the cached features to the target features.

7. The diffusion model inference acceleration method according to claim 1, characterized in that, Based on the preset early-stage mapping protection parameters, the set of main key time steps is determined from the target number of key time steps, as shown in equation (7): （7）； in, This represents the set of key time steps. This indicates the number of key time steps in the target sequence. A key time step; This indicates the preset parameters for protecting the initial layout. This represents the total number of critical time steps.

8. An electronic device, characterized in that, The device includes a processor and a memory, the memory storing at least one instruction and at least one program, the at least one instruction and the at least one program being loaded and executed by the processor to implement the diffusion model inference acceleration method as described in any one of claims 1 to 7.

9. A computer storage medium, characterized in that, The computer storage medium stores at least one instruction and at least one program, which are loaded and executed by a processor to implement the diffusion model inference acceleration method as described in any one of claims 1 to 7.

Images