Diffusion model quantification method and system based on Pareto search and decoupling operator

Abstract

The invention provides a Pareto search and decoupling operator-based diffusion model quantification method and system, and the method comprises the steps: carrying out the full-time-sequence sensitivity analysis of a pre-trained diffusion model, and obtaining the sensitivity weight of each network layer in the diffusion model at each time step; constructing a decoupling candidate set based on bit width and operator configuration; obtaining the real reasoning delay of the decoupling candidate set, and constructing a delay lookup table; based on the sensitivity weight and the delay lookup table, executing Pareto search on the decoupling candidate set to obtain a full-time-sequence optimal quantization strategy path; and performing quantization processing on the pre-trained diffusion model based on the optimal quantization strategy path of the full time sequence to obtain a quantized diffusion model. The quantitative diffusion model can effectively support generation tasks of data such as images and videos, resource waste is avoided, and structure collapse in the data transmission and generation process is reduced to the maximum extent.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and computer vision technology, and in particular to a diffusion model quantization method and system based on Pareto search and decoupling operators. Background Technology

[0002] With the widespread application of diffusion transformers (DiTs) in image and video generation (such as Flux and CogVideoX), their large number of parameters and multi-step iterative generation mechanism have brought about huge computational overhead and memory consumption. To achieve edge deployment, model quantization has become a key technology. However, existing post-training quantization (PTQ) techniques for diffusion models have the following significant drawbacks: 1. Static quantization strategies are unsuitable for dynamic generation processes: Existing quantization methods (such as SVDQuant and Q-DiT) typically employ a "static" strategy, forcing the model to use a uniform bit width (e.g., W4A4) or a uniform operator configuration (e.g., always enabling rotation) throughout all timesteps of the denoising process. This method ignores the temporal non-uniformity of the diffusion generation process: the early stages of generation (graphing phase) are extremely sensitive to outliers and structural information, requiring high precision or strong operators for protection; while the later stages of generation (texturing phase) have sparse activation values ​​and high tolerance for noise, resulting in significant computational redundancy. The static strategy leads to the contradiction of "insufficient precision in critical steps and wasted computational power in redundant steps."

[0003] 2. Decoupling of Search Space and Hardware Characteristics: Existing mixed-precision search methods typically only use "bit width" as the search variable, ignoring the impact of quantization operators (such as whether Hadamard rotation is enabled or whether rearrangement is performed) on model performance. In reality, the strength of quantization operators is coupled with time-step sensitivity (for example, strong rotation operators allow the use of low bits in the early stages, but increase computational overhead). Furthermore, existing methods often use theoretical computational cost (bit-ops) as the cost function for the search, which fails to reflect the operator switching overhead and memory access latency on real hardware, resulting in the searched strategy not showing significant acceleration in actual deployment.

[0004] 3. Lack of global consideration for error accumulation: Existing layer-wise or block-wise quantization methods (such as CLQ, S2Q-VDiT) mainly focus on the current local reconstruction error. However, the essence of diffusion generation is solving ordinary differential equations (ODEs). Small quantization errors in early steps will cause trajectory deviation and accumulate over time. Optimizing only the single-step error while ignoring the trajectory consistency throughout time will lead to structural collapse or semantic drift in the generated image.

[0005] A literature search of existing technologies revealed a Chinese patent with publication number CN117892792A, which proposes a mixed-precision quantization method for generating images using a diffusion model. This method allocates quantization bit widths to different layers based on their quantization sensitivity, thereby accelerating the generation of the diffusion model more rationally and efficiently. While this application employs dynamic resource allocation, it lacks a global consideration of error accumulation, making the generated structure prone to collapse at low bit depths. Therefore, there is an urgent need for a diffusion model quantization method and system that can implement a dynamic quantization strategy while maintaining semantic consistency of the generated content even at low average bit depths. Summary of the Invention

[0006] To address the shortcomings of existing technologies, the purpose of this application is to provide a diffusion model quantization method and system based on Pareto search and decoupling operators.

[0007] According to the first aspect of this application, a diffusion model quantization method based on Pareto search and decoupling operators is provided, comprising: A full-time sensitivity analysis was performed on the pre-trained diffusion model to obtain the sensitivity weights of each network layer in the diffusion model at each time step. Construct a decoupling candidate set based on bit width and operator configuration; Obtain the actual inference delay of the decoupling candidate set and construct a delay lookup table; Based on the sensitivity weights and the delay lookup table, a Pareto search is performed on the decoupling candidate set to obtain the optimal quantization strategy path for the entire time series. The pre-trained diffusion model is quantized using the optimal quantization strategy path based on the full time series, resulting in a quantized diffusion model.

[0008] Optionally, the step of performing a full-time sensitivity analysis on the pre-trained diffusion model to obtain the sensitivity weights of each network layer in the diffusion model at each time step includes: A calibration dataset is constructed and input into a pre-trained diffusion model to obtain Fisher information of each network layer in the pre-trained diffusion model at each time step of the diffusion process. The Fisher information is then used as a quantification index of the sensitivity of the corresponding network layer at each time step. For each network layer, all time steps are sorted from high to low according to the sensitivity index. Each time step is assigned a sensitivity weight based on the relative size of the sensitivity index. The time step with the higher sensitivity index corresponds to a higher sensitivity weight value. The sensitivity weights of all time steps are normalized so that the sum of the sensitivity weights of all time steps in a network layer is 1.

[0009] Optionally, the construction of the decoupling candidate set based on bit width and operator configuration includes: A globally unified decoupling candidate set is constructed. This set provides optional candidate strategies for all network layers and time steps of the pre-trained diffusion model. Each candidate strategy consists of a bit-width parameter and an operator configuration parameter. The specific details of each candidate strategy are as follows: High-fidelity mode strategy: The bit width parameter is set to the preset high bit width, and the operator configuration parameter is set to enable Hadamard rotation; Standard mode strategy: The bit width parameter is set to the preset low bit width, and the operator configuration parameter is set to enable Hadamard rotation; High-speed mode strategy: The bit width parameter is set to the preset low bit width, and the operator configuration parameter is set to disable Hadamard rotation.

[0010] Optionally, obtaining the true inference latency of the decoupling candidate set and constructing a latency lookup table includes: A target hardware platform is selected. On the target hardware platform, for all candidate strategies in the decoupled candidate set, the real inference latency of each candidate strategy on each network layer of the pre-trained diffusion model is tested, and a latency lookup table is constructed based on the test results. The latency lookup table uses the network layer and the candidate strategy as index values ​​to store the corresponding real inference latency data.

[0011] Optionally, the step of performing a Pareto search on the decoupling candidate set based on the sensitivity weights and the delay lookup table to obtain the optimal quantization strategy path for the entire time series includes: Construct a search tree, wherein each leaf node in the search tree corresponds to a combination unit, and the combination unit includes a network layer in the pre-trained diffusion model, a time step in which the network layer participates in inference computation, and a candidate strategy adopted by the network layer at the corresponding time step. A multi-objective integrated loss function is constructed based on the sensitivity weights and the delay lookup table. By combining a multi-objective comprehensive loss function, a Pareto grid search is performed on the candidate policies in the search tree to obtain the optimal quantization policy path for the entire time series.

[0012] Optionally, the expression for the multi-objective integrated loss function is as follows: in, This represents the loss of the i-th network layer at time step t; This represents the sensitivity weight of the i-th network layer at time step t; This indicates that the actual inference latency data is stored in the latency lookup table, with the corresponding candidate policy at the i-th network layer and time step t as the index value. These are the preset trade-off coefficients.

[0013] Optionally, the step of combining a multi-objective comprehensive loss function to perform Pareto search on candidate policies in the search tree to obtain the optimal quantization policy path for the entire time series includes: Along the diffusion model inference timeline, with time steps as the iteration dimension, perform the following operations step by step: Extract all leaf nodes of each network layer at time step t, and use them as the node set at time step t. According to the rule that each network layer selects only one leaf node corresponding to it at time step, select one corresponding leaf node for each network layer from the node set at time step t. Combine the selected leaf nodes, and each combination result is a set of candidate strategy combinations at time step t. Generate all paths at time step t: If t=1: each combination of candidate strategies at time step t is considered as a path; If t>1: retrieve all the basic paths selected in the previous time step, and concatenate each basic path with each group of candidate strategies in time step t. The concatenation result is a path in time step t. At time step t, the multi-objective integrated loss function is called to obtain the loss of each path. The loss of each path is the sum of the cumulative losses of all network layers from time step 1 to time step t. Pareto pruning is performed on all paths to select the K paths with the least loss as the base paths for the iteration at time step t+1. When the iteration reaches t = total time step, the iteration is terminated and the path with the minimum loss is taken as the optimal quantization strategy path.

[0014] Optionally, the optimal quantization strategy path based on full-time series is used to quantize the pre-trained diffusion model to obtain a quantized diffusion model, including: During the inference process of the pre-trained diffusion model, the candidate policy combination based on the optimal quantization policy path at different time steps is used to quantize the bit width and operator configuration of each network layer at that time step, so as to realize the quantization processing of the pre-trained diffusion model.

[0015] According to a second aspect of this application, a diffusion model quantization system based on Pareto search and decoupling operators is provided, comprising: The weight analysis module is used to perform full-time sensitivity analysis on the pre-trained diffusion model to obtain the sensitivity weight of each network layer in the diffusion model at each time step. The decoupling module is used to construct a set of decoupling candidates based on bit width and operator configuration; The delay acquisition module is used to acquire the actual inference delay of the decoupling candidate set and construct a delay lookup table; The path generation module is used to perform Pareto search on the decoupling candidate set based on the sensitivity weights and the delay lookup table to obtain the optimal quantization strategy path for the entire time series. The quantization module is used to quantize the pre-trained diffusion model based on the optimal quantization strategy path across the entire time series, resulting in a quantized diffusion model.

[0016] According to a third aspect of this application, an image generation method is provided, comprising: Determine the pre-trained FLUX text-to-image model; The pre-trained FLUX text-to-graph model is quantized using any of the diffusion model quantization methods based on Pareto search and decoupling operators provided in the first aspect of this application, to determine the quantized FLUX text-to-graph model. The preset image-generated text is input into the quantized FLUX text-generated image model to determine the generated image.

[0017] This application provides a diffusion model quantization method based on Pareto search and decoupling operators. It performs full-time sensitivity analysis on the diffusion model, enabling on-demand allocation of computing resources along the time axis, thus breaking through the performance limits of static quantization techniques and achieving superior performance. By constructing a decoupling candidate set based on bit width and operator configuration, it overcomes the limitations of traditional mixed-precision search, which only adjusts bit width and ignores operator overhead. Furthermore, Pareto search based on a delayed lookup table achieves a coordinated balance of multiple objectives, maintaining semantic consistency and structural integrity of the generated content even with extremely low average bit depth.

[0018] Other technical effects resulting from the additional features will be further illustrated in the corresponding embodiments. Attached Figure Description

[0019] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart of a diffusion model quantization method in one embodiment of this application; Figure 2 This is a flowchart of Pareto pruning search and dynamic configuration generation in one embodiment of this application; Figure 3 This is a schematic diagram of a diffusion model quantization system in one embodiment of this application. Detailed Implementation

[0020] The present application will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application, and these all fall within the protection scope of the present application. Parts not described in detail in the following embodiments can be implemented using existing technology.

[0021] It should be noted that all information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with relevant regulations.

[0022] With the widespread application of diffusion transformers in image and video generation, their large number of parameters and multi-step iterative generation mechanism bring huge computational overhead and memory consumption. To achieve edge deployment, model quantization has become a key technology. However, existing post-training quantization techniques for diffusion models have the following significant drawbacks: static quantization strategies cannot adapt to dynamic generation processes; the search target is disconnected from the hardware, resulting in insignificant acceleration effects of the searched strategies in actual deployment; and there is a lack of global consideration for error accumulation, leading to structural collapse or semantic drift in the generated images. Based on these problems, this application provides a diffusion model quantization method based on Pareto search and decoupling operators to solve the aforementioned issues.

[0023] Reference Figure 1 As shown, this application provides a diffusion model quantization method based on Pareto search and decoupling operators, including: S1. Perform a full-time sensitivity analysis on the pre-trained diffusion model to obtain the sensitivity weights of each network layer in the diffusion model at each time step. S2. Construct a decoupling candidate set based on bit width and operator configuration; S3. Obtain the actual inference delay of the decoupling candidate set and construct a delay lookup table; S4. Based on the sensitivity weight and the delay lookup table, perform Pareto search on the decoupling candidate set to obtain the optimal quantization strategy path for the entire time series. S5. The pre-trained diffusion model is quantized based on the optimal quantization strategy path of the full time series to obtain the quantized diffusion model.

[0024] The embodiments described above in this application perform full-time sensitivity analysis on the diffusion model, enabling on-demand allocation of computing resources along the time axis, thereby breaking through the performance limit of static quantization technology and achieving better running performance; constructing a decoupled candidate set based on bit width and operator configuration can overcome the limitations of traditional mixed-precision search that only adjusts bit width and ignores operator overhead; Pareto search based on delayed lookup table can achieve a coordinated balance of multiple objectives, maintaining semantic consistency and structural integrity of generated content under extremely low average bit depth.

[0025] The quantization diffusion model proposed in this application can effectively support generation tasks of data such as images and videos. Through dynamic resource allocation, operator-bit-width coupling, and Pareto search, it can avoid resource waste and minimize structural collapse during data transmission and generation. This quantized diffusion model can independently adapt to scenarios such as ultra-high-definition video encoding and local image editing on mobile terminals, including specific needs such as portrait generation and landscape image quality optimization. Leveraging its own quantization advantages, it eliminates dependence on cloud computing power and fully utilizes the adaptability and performance advantages of low-bit quantization technology in terminal applications.

[0026] In some specific embodiments of this application, performing a full-time sensitivity analysis on the pre-trained diffusion model to obtain the sensitivity weights of each network layer in the diffusion model at each time step may further include: S11. Construct a calibration dataset, which includes N calibration input samples and their corresponding benchmark true values; S12. Input the calibration dataset into the pre-trained diffusion model to obtain the Fisher information of each network layer in the pre-trained diffusion model at each time step of the diffusion process, and use the Fisher information as the sensitivity index of the corresponding network layer at each time step. S13. For each network layer, sort all time steps from high to low according to the sensitivity index, and assign sensitivity weights to each time step according to the relative size of the sensitivity index. The time step with the higher sensitivity index corresponds to a higher sensitivity weight value. Normalize the sensitivity weights of all time steps so that the sum of the sensitivity weights of all time steps in a network layer is 1.

[0027] For example, this step is used to construct a full-time sensitivity profile, that is, to perform a full-time sensitivity analysis on the pre-trained diffusion model: using a calibration dataset, the Fisher information of each network layer at each inference time step t is calculated as an indicator of the contribution of that network layer to the quality of the generated image / video at the current inference time step. The Fisher information is obtained as follows: The calibration input samples from the calibration dataset are input into the pre-trained diffusion model. For each network layer of the pre-trained diffusion model at each time step t in the diffusion process, the baseline true value in the calibration dataset that corresponds one-to-one with the calibration input sample is used as a reference. The Fisher information matrix calculation method in the existing technology is adopted, and the sample covariance matrix of the log-likelihood gradient is used to obtain the Fisher information of each network layer at time step t. The sensitivity index and the sensitivity weight value can be linearly related.

[0028] The embodiments described above in this application employ dynamic resource allocation technology based on full-time sensitivity profiling, which enables on-demand allocation of computing resources along the time axis and solves the problem of performance limitations caused by "insufficient accuracy of key steps and waste of computing power in redundant steps" in existing static quantization methods.

[0029] In some specific embodiments of this application, constructing a decoupling candidate set based on bit width and operator configuration may further include: A globally unified decoupling candidate set is constructed. This set provides optional candidate strategies for all network layers and time steps of the pre-trained diffusion model. Each candidate strategy consists of a bit-width parameter and an operator configuration parameter. The specific details of each candidate strategy are as follows: High-fidelity mode strategy: The bit width parameter is set to the preset high bit width, and the operator configuration parameter is set to enable Hadamard rotation; Standard mode strategy: The bit width parameter is set to the preset low bit width, and the operator configuration parameter is set to enable Hadamard rotation; High-speed mode strategy: The bit width parameter is set to the preset low bit width, and the operator configuration parameter is set to disable Hadamard rotation.

[0030] For example, this step is used to construct a decoupled quantization operator space. Unlike traditional methods that only search for bit width, this application constructs a decoupled candidate set containing both bit width and operator configuration within the decoupled quantization operator space. For each time step of the network layer, candidate strategies include, but are not limited to: High-fidelity mode strategy (Type A): High bit width (e.g., W8), enable Hadamard Rotation. This strategy can be used for high-sensitivity time steps to suppress outliers. Standard mode strategy (Type B, Type D): Low bit width (such as W4 / W3 / W2), enable Hadamard rotation, this strategy is used as the baseline configuration; Extreme speed mode strategy (Type C): low bit width (such as W4 / W3 / W2), no-rotation. This strategy is used for robust time steps and further speeds up the process by reducing computation.

[0031] This step provides a variety of atomic operations with a "precision-speed" trade-off for each time step, with each atomic operation corresponding to a fixed set of bit width parameters and operator configuration parameters.

[0032] The above embodiments of this application employ a candidate strategy space construction technique that decouples bit width and rotation operator, which can achieve optimal coupling between the strength of quantization operator and time step sensitivity (such as using strong operators to preserve structure in the early stage and turning off rotation to change speed in the late stage). This solves the problem that traditional mixed-precision search only adjusts bit width and ignores operator overhead, and cannot tap the ultimate acceleration potential of hardware.

[0033] In some specific embodiments of this application, obtaining the actual inference latency of the decoupling candidate set and constructing a latency lookup table may further include: Select a target hardware platform. On the target hardware platform, for all candidate strategies in the decoupled candidate set, test the real inference latency of each candidate strategy on each network layer of the pre-trained diffusion model, and construct a latency lookup table based on the test results. The latency lookup table uses network layer and candidate strategy as index values ​​to store the corresponding real inference latency data.

[0034] In the embodiments described above, this step is used to establish a Latency Lookup Table (LUT). On the target hardware platform (such as a cloud computing platform, edge computing device, or embedded terminal device), the actual inference latency (Measured Latency) of various candidate strategies in step S2 on different network layer types is pre-tested to construct the latency lookup table. This step discards inaccurate theoretical bit count metrics, ensuring that the search results can be translated into actual acceleration.

[0035] In some specific embodiments of this application, a Pareto search is performed on the decoupling candidate set based on sensitivity weights and a delay lookup table to obtain the optimal quantization strategy path for the entire time series, which may further include: S41. Construct a search tree. Each leaf node in the search tree corresponds to a combination unit. The combination unit includes a network layer in the pre-trained diffusion model, a time step in which the network layer participates in inference computation, and a candidate strategy adopted by the network layer at the corresponding time step. S42. Construct a multi-objective integrated loss function based on sensitivity weights and a delay lookup table; S43. Combining the multi-objective integrated loss function, Pareto grid search is performed on the candidate policies in the search tree to obtain the optimal quantization policy path for the entire time series.

[0036] In some specific embodiments of this application, the expression for the multi-objective integrated loss function is as follows: in, This represents the loss of the i-th network layer at time step t; This represents the sensitivity weight of the i-th network layer at time step t; This indicates that the actual inference latency data is stored in the latency lookup table, with the corresponding candidate policy at the i-th network layer and time step t as the index value. These are the preset trade-off coefficients.

[0037] In some specific embodiments of this application, combining a multi-objective comprehensive loss function, Pareto search is performed on candidate policies in the search tree to obtain the optimal quantization policy path for the entire time series, which may further include: Along the diffusion model inference timeline, with time steps as the iteration dimension, perform the following operations step by step: S431. Extract all leaf nodes of each network layer at time step t, and use them as the node set at time step t. According to the rule that each network layer selects only one leaf node corresponding to it at time step, select one corresponding leaf node for each network layer from the node set at time step t. Combine the selected leaf nodes, and each combination result is a set of candidate strategy combinations at time step t. S432. Generate all paths at time step t: If t=1: each combination of candidate strategies at time step t is considered as a path; If t>1: retrieve all the basic paths selected in the previous time step, and concatenate each basic path with each candidate strategy combination in time step t. The result of each concatenation is a path in time step t. S433. At time step t, call the multi-objective integrated loss function to obtain the loss of each path. The loss of each path is the sum of the cumulative losses of all network layers from time step 1 to time step t. Perform Pareto pruning on all paths and select the K paths with the least loss as the basic paths for the iteration at time step t+1. S434. When the iteration reaches t = total time step, terminate the iteration and take the path with the minimum loss as the optimal quantization strategy path.

[0038] For example, the method for generating all paths at time step t (t>1) is as follows: All base paths at time step t-1 are concatenated temporally with all candidate strategy combinations at time step t. That is, each base path is concatenated temporally with each candidate strategy combination at time step t, and each independent concatenation result corresponds to a path at time step t. During Pareto pruning, K represents the preset number of paths to filter, which can be between 5 and 20. The above steps are used to perform Pareto lattice search (PPS), modeling the quantization strategy search as a path planning problem on the time axis. The steps include: State definition: Each node in the search tree represents a specific quantization configuration (i.e., candidate strategy) of a certain network layer at a certain time step.

[0039] Objective function: Construct a loss function that includes time-series compensation: in, It incorporates time-sensitivity weights to compensate for error accumulation in the TeacherForcing mode; A user-defined tradeoff coefficient.

[0040] Search process: Using dynamic programming, a Pareto front is maintained. At each step, the K most cost-effective paths are retained, and finally, the optimal quantization strategy path for the entire time series is output.

[0041] In some specific embodiments of this application, the pre-trained diffusion model is quantized based on the optimal quantization strategy path throughout the time series to obtain a quantized diffusion model, which may further include: During the inference process of the pre-trained diffusion model, the candidate policy combination based on the optimal quantization policy path at different time steps is used to quantize the bit width and operator configuration of each network layer at that time step, so as to realize the quantization processing of the pre-trained diffusion model.

[0042] For example, step S5 is used for adaptive quantization inference: During the inference phase, the diffusion model dynamically switches the quantization operator and bit width configuration at different time steps according to the optimal quantization strategy path output in step S4 to complete the generation of images or videos.

[0043] This application provides a diffusion model quantization method based on Pareto search and decoupling operators, which specifically addresses the following technical problems: 1. Solve the problem of temporal resource mismatch: By constructing a full temporal sensitivity profile, the "construction period" and "texturing period" in the generation process are automatically identified, and high / low bit width strategies are dynamically allocated accordingly, breaking the performance ceiling of static quantization; 2. Achieve Pareto optimality through hardware and software collaboration: Decouple the quantization bit width from the operator configuration (such as rotation switch, rearrangement strategy) to construct a candidate strategy pool, and search based on the real hardware latency (Latency LUT) to ensure that the obtained quantization strategy achieves Pareto optimality in the "image quality-speed" tradeoff.

[0044] 3. Suppressing the accumulation of error trajectories: Introducing a global search target with temporal compensation, and explicitly penalizing policy paths that may lead to trajectory deviation during the search process, thereby maintaining the semantic consistency and structural integrity of the generated content at extremely low average bit counts (such as W3A3).

[0045] The embodiments described above employ Pareto search technology based on real delay lookup table (LUT) and trajectory error compensation, which can achieve Pareto optimality of image quality and inference speed at extremely low average bit depths (such as W3A3). This solves the problems that theoretical computational cost (Bit-ops) cannot reflect the actual speedup ratio and that the generated trajectory deviates due to neglecting error accumulation in Teacher Forcing mode.

[0046] This application enables high-fidelity, high-speed adaptive inference of the diffusion model at extremely low bit depths, solving problems such as rigid static quantization strategies, disconnect between search targets and hardware, and structural collapse in low-bit environments in existing technologies. Short videos and images shot by ordinary users often suffer from quality shortcomings such as shaky footage, insufficient clarity, and limited shooting angles, making it difficult to directly produce ideal finished products. The quantization-diffusion model proposed in this application can maintain semantic consistency and structural integrity of generated content under the limited computing power constraints of mobile devices, achieving high-quality reproduction of high-definition short videos and images—whether it's sharp mountain landscapes, natural-looking close-ups of people, or clear-textured food images.

[0047] The following examples will be used to further illustrate this application in order to better understand the above-mentioned technical solutions. It should be understood that the following are only some examples and are not intended to limit this application.

[0048] Application Example 1: W3A3 Adaptive Quantization Inference Based on Flux.1-dev Model This application example applies the method of this application to a Transformer-based text graph model (Flux.1-dev, 12-bit parameter count). The goal is to reduce the average bit width to approximately 3 bits while maintaining lossless FID, and to achieve significant acceleration on the NVIDIA RTX 4090. (Refer to...) Figure 2 As shown, the steps include: Step 1: Temporal sensitivity and operator space construction Sensitivity analysis: Full-precision inference was run using 16 calibrated input samples in the Prompt. Fisher information for each network layer was calculated at 50 time steps.

[0049] Operator space definition: Define four candidate strategies for each network layer: Type A: W8A8 + Enable Rotation (Hadamard Rotation).

[0050] Type B: W4A4+ Enable rotation.

[0051] Type C: W4A4+ No-Rotation.

[0052] Type D: W2A2+ Enable rotation.

[0053] Step 2: Latency Profiling The kernel kernel time of the above four strategies was measured on the target hardware platform RTX 4090.

[0054] Construct a Latency LUT table.

[0055] Step 3: Perform a PPS search Set the velocity weights ($\lambda$) in the objective function, run a Pareto lattice search, and obtain the policy table (i.e., the optimal quantization policy path). Step 4: Adaptive Inference Verification Load the search-obtained Schedule Table for inference, setting the equivalent bit width to 3.2 bits.

[0056] In some specific embodiments of this application, a diffusion model quantization method based on Pareto search and decoupling operators can be applied to the FLUX text-generated image model. Specifically, an image generation method includes: Step 1: Determine the pre-trained FLUX text-to-image model; Step 2: Use the diffusion model quantization method based on Pareto search and decoupling operator in any of the above embodiments to quantize the pre-trained FLUX text graph model and determine the quantized FLUX text graph model. Step 3: Input the preset image-generated text into the quantized FLUX text-generated image model to determine the generated image.

[0057] In some specific embodiments of this application, a diffusion model quantization method based on Pareto search and decoupling operators can be applied to the Wan video generation model. Specifically, a video generation method includes: Step 1: Determine the pre-trained Wan video generation model; Step 2: Use the diffusion model quantization method based on Pareto search and decoupling operator in any of the above embodiments to quantize the pre-trained Wan video generation model and determine the quantized Wan video generation model. Step 3: Input the preset video generation text into the quantized Wan video generation model to determine the generated video.

[0058] Based on the same inventive concept, another embodiment of this application provides a diffusion model quantization system based on Pareto search and decoupling operators, referring to... Figure 3 As shown, the diffusion model quantization system 100 includes: The weight analysis module 110 is used to perform full-time sensitivity analysis on the pre-trained diffusion model to obtain the sensitivity weight of each network layer in the diffusion model at each time step. Decoupling module 120 is used to construct a decoupling candidate set based on bit width and operator configuration; The delay acquisition module 130 is used to acquire the actual inference delay of the decoupling candidate set and construct a delay lookup table. The path generation module 140 is used to perform Pareto search on the decoupling candidate set based on sensitivity weights and a delay lookup table to obtain the optimal quantization strategy path for the entire time series. The quantization module 150 is used to quantize the pre-trained diffusion model based on the optimal quantization strategy path of the entire time series, so as to obtain the quantized diffusion model.

[0059] It should be noted that the modules in the diffusion model quantization system based on Pareto search and decoupling operators provided in the above embodiments of this application correspond to the steps of the diffusion model quantization method based on Pareto search and decoupling operators in any of the above embodiments. Those skilled in the art can refer to the step features of the diffusion model quantization method based on Pareto search and decoupling operators to implement the corresponding modules in the diffusion model quantization system based on Pareto search and decoupling operators, which will not be elaborated here.

[0060] In another embodiment of this application, an electronic device is also provided, including a memory and a processor; the memory is used to store program instructions; the processor is used to call the program instructions stored in the memory and execute the steps of the diffusion model quantization method based on Pareto search and decoupling operators as described above according to the obtained program instructions.

[0061] Optionally, the memory is used to store programs; the memory may include volatile memory, such as random-access memory (RAM), such as static random-access memory (SRAM), double data rate synchronous dynamic random-access memory (DDR SDRAM), etc.; the memory may also include non-volatile memory, such as flash memory. The memory is used to store computer programs (such as application programs and functional modules that implement the above methods), computer instructions, etc., and the aforementioned computer programs and computer instructions can be partitioned and stored in one or more memories. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by the processor.

[0062] The aforementioned computer programs, computer instructions, etc., can be stored in partitions within one or more memory locations. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by a processor.

[0063] A processor is used to execute a computer program stored in memory to implement the various steps of the methods involved in the above embodiments. For details, please refer to the relevant descriptions in the preceding method embodiments.

[0064] The processor and memory can be separate structures or integrated structures. When the processor and memory are separate structures, they can be coupled together via a bus.

[0065] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0066] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0067] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0068] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0069] The preferred features in the above embodiments can be used individually in any embodiment, or in any combination thereof, provided they do not conflict with each other. Furthermore, parts not described in detail in the embodiments can be implemented using existing technologies.

[0070] The foregoing has described some specific embodiments of this application. It should be understood that this application is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the substantive content of this application. The above-described preferred features can be used in any combination without conflict.

Claims

1. A quantization method for diffusion models based on Pareto search and decoupling operators, characterized in that, include: A full-time sensitivity analysis was performed on the pre-trained diffusion model to obtain the sensitivity weights of each network layer in the diffusion model at each time step. Construct a decoupling candidate set based on bit width and operator configuration; Obtain the actual inference delay of the decoupling candidate set and construct a delay lookup table; Based on the sensitivity weights and the delay lookup table, a Pareto search is performed on the decoupling candidate set to obtain the optimal quantization strategy path for the entire time series. The pre-trained diffusion model is quantized using the optimal quantization strategy path based on the full time series, resulting in a quantized diffusion model.

2. The diffusion model quantization method based on Pareto search and decoupling operators according to claim 1, characterized in that, The full-time sensitivity analysis of the pre-trained diffusion model yields the sensitivity weights of each network layer at each time step, including: A calibration dataset is constructed and input into a pre-trained diffusion model to obtain Fisher information of each network layer in the pre-trained diffusion model at each time step of the diffusion process. The Fisher information is then used as a quantification index of the sensitivity of the corresponding network layer at each time step. For each network layer, all time steps are sorted from high to low according to the sensitivity index. Each time step is assigned a sensitivity weight based on the relative size of the sensitivity index. The time step with the higher sensitivity index corresponds to a higher sensitivity weight value. The sensitivity weights of all time steps are normalized so that the sum of the sensitivity weights of all time steps in a network layer is 1.

3. The diffusion model quantization method based on Pareto search and decoupling operators according to claim 1, characterized in that, The construction of the decoupling candidate set based on bit width and operator configuration includes: A globally unified decoupling candidate set is constructed. This set provides optional candidate strategies for all network layers and time steps of the pre-trained diffusion model. Each candidate strategy consists of a bit-width parameter and an operator configuration parameter. The specific details of each candidate strategy are as follows: High-fidelity mode strategy: The bit width parameter is set to the preset high bit width, and the operator configuration parameter is set to enable Hadamard rotation; Standard mode strategy: The bit width parameter is set to the preset low bit width, and the operator configuration parameter is set to enable Hadamard rotation; High-speed mode strategy: The bit width parameter is set to the preset low bit width, and the operator configuration parameter is set to disable Hadamard rotation.

4. The diffusion model quantization method based on Pareto search and decoupling operators according to claim 3, characterized in that, The step of obtaining the true inference delay of the decoupling candidate set and constructing a delay lookup table includes: A target hardware platform is selected. On the target hardware platform, for all candidate strategies in the decoupled candidate set, the real inference latency of each candidate strategy on each network layer of the pre-trained diffusion model is tested, and a latency lookup table is constructed based on the test results. The latency lookup table uses the network layer and the candidate strategy as index values ​​to store the corresponding real inference latency data.

5. The diffusion model quantization method based on Pareto search and decoupling operators according to claim 3, characterized in that, The step of performing a Pareto search on the decoupling candidate set based on the sensitivity weights and the delay lookup table to obtain the optimal quantization strategy path for the entire time series includes: Construct a search tree, wherein each leaf node in the search tree corresponds to a combination unit, and the combination unit includes a network layer in the pre-trained diffusion model, a time step in which the network layer participates in inference computation, and a candidate strategy adopted by the network layer at the corresponding time step. A multi-objective integrated loss function is constructed based on the sensitivity weights and the delay lookup table. By combining a multi-objective comprehensive loss function, a Pareto grid search is performed on the candidate policies in the search tree to obtain the optimal quantization policy path for the entire time series.

6. The diffusion model quantization method based on Pareto search and decoupling operators according to claim 5, characterized in that, The expression for the multi-objective integrated loss function is as follows: ； in, This represents the loss of the i-th network layer at time step t; This represents the sensitivity weight of the i-th network layer at time step t; This indicates that the actual inference latency data is stored in the latency lookup table, with the corresponding candidate policy at the i-th network layer and time step t as the index value. This is a preset tradeoff coefficient.

7. The diffusion model quantization method based on Pareto search and decoupling operators according to claim 5, characterized in that, The step of combining a multi-objective comprehensive loss function to perform Pareto search on candidate policies in the search tree to obtain the optimal quantization policy path for the entire time series includes: Along the diffusion model inference timeline, with time steps as the iteration dimension, perform the following operations step by step: Extract all leaf nodes of each network layer at time step t, and use them as the node set at time step t. According to the rule that each network layer selects only one leaf node corresponding to it at time step, select one corresponding leaf node for each network layer from the node set at time step t. Combine the selected leaf nodes, and each combination result is a set of candidate strategy combinations at time step t. Generate all paths at time step t: If t=1: each combination of candidate strategies at time step t is considered as a path; If t>1: retrieve all the basic paths selected in the previous time step, and concatenate each basic path with each group of candidate strategies in time step t. The concatenation result is a path in time step t. At time step t, the multi-objective integrated loss function is called to obtain the loss of each path. The loss of each path is the sum of the cumulative losses of all network layers from time step 1 to time step t. Pareto pruning is performed on all paths to select the K paths with the least loss as the base paths for the iteration at time step t+1. When the iteration reaches t = total time step, the iteration is terminated and the path with the minimum loss is taken as the optimal quantization strategy path.

8. The diffusion model quantization method based on Pareto search and decoupling operators according to claim 7, characterized in that, The optimal quantization strategy path based on full-time series quantization processes the pre-trained diffusion model to obtain a quantized diffusion model, including: During the inference process of the pre-trained diffusion model, the candidate policy combination based on the optimal quantization policy path at different time steps is used to quantize the bit width and operator configuration of each network layer at that time step, so as to realize the quantization processing of the pre-trained diffusion model.

9. A diffusion model quantization system based on Pareto search and decoupling operators, characterized in that, include: The weight analysis module is used to perform full-time sensitivity analysis on the pre-trained diffusion model to obtain the sensitivity weight of each network layer in the diffusion model at each time step. The decoupling module is used to construct a set of decoupling candidates based on bit width and operator configuration; The delay acquisition module is used to acquire the actual inference delay of the decoupling candidate set and construct a delay lookup table; The path generation module is used to perform Pareto search on the decoupling candidate set based on the sensitivity weights and the delay lookup table to obtain the optimal quantization strategy path for the entire time series. The quantization module is used to quantize the pre-trained diffusion model based on the optimal quantization strategy path across the entire time series, resulting in a quantized diffusion model.

10. An image generation method, characterized in that, include: Determine the pre-trained FLUX text-to-image model; The pre-trained FLUX text-to-graph model is quantized using any one of the diffusion model quantization methods based on Pareto search and decoupling operators in claims 1 to 8, thereby determining the quantized FLUX text-to-graph model. The preset image-generated text is input into the quantized FLUX text-generated image model to determine the generated image.

Images