Diffusion model space-time mixing precision quantification method and system based on tree search

Abstract

The invention provides a diffusion model space-time mixing precision quantification method and system based on tree search, and the method comprises the steps: constructing a space search tree, and carrying out the aggregation of the space search tree, and obtaining the static weight precision and reference activation precision of a DiT model; discretizing the time dimension of the DiT model to generate a key time period; based on the reference activation precision, obtaining an average activation bit number and accumulated distortion of each key time period; constructing a time search tree, and aggregating the time search tree by using an average activation bit number and accumulated distortion to obtain an optimal scheduling path of the DiT model; and performing quantization processing on the pre-trained DiT model based on the static weight precision and the optimal scheduling path to obtain a quantized DiT model. The quantization DiT model can be applied to generation scenes of data such as images and videos, and distortion minimization of the data can be achieved under the condition of extremely low bit number budget through highly unified logic of spatial dimension and time.

Description

Technical Field

[0001] This application relates to the fields of deep learning model compression and automated machine learning (AutoML) technology, and in particular to a spatiotemporal hybrid precision quantization method and system for diffusion models based on tree search. Background Technology

[0002] Mixed-precision quantization (MPQ) is a key technique for reducing the inference cost of diffusion models. However, existing quantization search methods for diffusion transformer (DiT) models suffer from the following significant dimensionality fragmentation problem: 1. Low spatial search efficiency: DiT models have numerous layers, leading to an exponential explosion in the traditional search space. Although methods based on integer programming and genetic algorithms have solved the spatial level search problem, they cannot fully explore the solution space, resulting in a low upper limit for mixed accuracy.

[0003] 2. Coarse Time-Dimensional Strategy: The generation process of the diffusion model involves dozens to hundreds of time steps. Existing methods either use the same quantization configuration for all time steps (static quantization), resulting in wasted computing power; or they use greedy algorithms or simple heuristic rules to allocate the precision of time steps, which cannot guarantee global optimization and is difficult to balance the constraints of "image quality" and "average bit count".

[0004] A literature search of existing technologies revealed a Chinese patent with publication number CN117892792A, which proposes a mixed-precision quantization method for generating diffusion models. This application, from the perspective of model quantization, allocates quantization bit widths to different layers based on their sensitivity to quantization, thereby accelerating the generation of diffusion models more reasonably and efficiently. However, the above method cannot unify time and space to achieve a high degree of consistency in algorithm logic.

[0005] Therefore, there is an urgent need for a diffusion model mixed precision quantization method that can perform unified diffusion model mixing in both time and space dimensions to minimize distortion. Summary of the Invention

[0006] To address the shortcomings of existing technologies, the purpose of this application is to provide a method and system for spatiotemporal precision quantization of diffusion models based on tree search.

[0007] According to the first aspect of this application, a spatiotemporal mixing precision quantization method for diffusion models based on tree search is provided, comprising: A spatial search tree is constructed, and the spatial search tree is aggregated to obtain the static weight accuracy and baseline activation accuracy of the DiT model; the DiT model is used to perform image or video generation tasks. The time dimension of the DiT model is discretized to generate key time periods; based on the baseline activation accuracy, the average number of activation bits and cumulative distortion for each key time period are obtained. Construct a time search tree, and aggregate the time search tree using the average number of activation bits and the cumulative distortion to obtain the optimal scheduling path for the DiT model; The pre-trained DiT model is quantized based on the static weight accuracy and optimal scheduling path to obtain the quantized DiT model.

[0008] Optionally, the construction of the spatial search tree and the aggregation of the spatial search tree to obtain the static weight accuracy and baseline activation accuracy of the DiT model include: Construct a candidate configuration set and a spatial search tree. The candidate configuration set includes several groups of quantized configurations. Each group of quantized configurations includes weight quantization precision and activation quantization precision. Each leaf node of the spatial search tree corresponds to a network layer in the DiT model. For each leaf node in the spatial search tree, the following operations are performed: traverse all quantizations in the candidate configuration set, calculate the average bit and quantization error of the leaf node under each quantization configuration, construct the spatial Pareto queue of the leaf node, the spatial Pareto queue contains several quantization units, each quantization unit includes a quantization configuration, its corresponding average bit and quantization error, and store the constructed spatial Pareto queue inside the corresponding leaf node. Aggregate the spatial search tree to obtain the spatial Pareto queue of the root node of the spatial search tree; The quantization configuration corresponding to the minimum quantization error when the average bit value is less than the preset weight bit threshold is selected from the spatial Pareto queue of the root node of the spatial search tree, and is used as the benchmark quantization configuration; the weight quantization precision in the benchmark quantization configuration is set to the static weight precision, and the activation quantization precision in the benchmark quantization configuration is set to the benchmark activation precision.

[0009] Optionally, the aggregation of the spatial search tree to obtain the spatial Pareto queue of the root node of the spatial search tree includes: Taking the level of the leaf node in the spatial search tree as the current level, iteratively perform the following operations: Create a new parent level above the current level; Traverse all nodes in the current level in topological order, select each pair of adjacent nodes as child nodes, combine the spatial Pareto queues stored in the selected child nodes by Cartesian product to generate a combined quantization queue. This combined quantization queue contains all possible quantization configuration combinations, and each quantization configuration combination constitutes a quantization unit. Perform spatial Pareto pruning on the combined quantization queue, select the Top-K quantization units at the Pareto front, and use them as a new spatial Pareto queue. Construct a parent node corresponding to the two child nodes, store the new spatial Pareto queue inside the parent node, and add the parent node to the previous level. Update the previous level to the current level. If the current level contains multiple nodes, continue iterating. When the current level contains only one node, terminate the iteration and set that node as the root node of the spatial search tree, and obtain the spatial Pareto queue of that root node.

[0010] Optionally, the discretization of the time dimension of the DiT model to generate key time periods; based on the baseline activation accuracy, obtaining the average number of activation bits and cumulative distortion for each key time period includes: The total number of inference steps in the DiT model inference process is divided into several non-overlapping key time periods; For each critical time period, the actual activation accuracy for that critical time period is determined based on the baseline activation accuracy. The expression for the actual activation accuracy is as follows: A real,j = A base + δ j δ j ∈Δ{-1,0,+1} Among them, A real,j A represents the actual activation precision during the j-th critical time period; base As the baseline activation accuracy; δ j Let be the precision offset for the j-th critical time period; Δ is the set of precision offsets. For each critical time period, the following operations are performed: traverse all precision offsets in the precision offset set, determine the corresponding actual activation precision based on the precision offset, and calculate the average number of activation bits and cumulative distortion when the DiT model performs inference with the actual activation precision during the critical time period.

[0011] Optionally, the construction of the time search tree, which involves aggregating the time search tree using the average number of activation bits and the cumulative distortion to obtain the optimal scheduling path for the DiT model, includes: Construct a time search tree, wherein each leaf node of the time search tree corresponds to a key time period; For each leaf node in the time search tree, perform the following operations: construct a time Pareto queue for that leaf node, the time Pareto queue containing several time units, each time unit including a precision offset, its corresponding average number of active bits and cumulative distortion, and store the constructed time Pareto queue inside the corresponding leaf node; Aggregate the time search tree to obtain the time Pareto queue of the root node of the time search tree; The optimal scheduling path is selected from the time Pareto queue of the root node of the time search tree, based on the precision offset corresponding to the condition that the average number of active bits is less than the preset active bit threshold and the cumulative distortion is minimized.

[0012] Optionally, the time search tree is aggregated to obtain the time Pareto queue of the root node of the time search tree, including: Using the level of the leaf node in the time search tree as the current level, iteratively perform the following operations: Create a new parent level above the current level; Traverse all nodes in the current level in topological order, selecting each pair of adjacent nodes as child nodes. Combine the time Pareto queues stored in the selected child nodes using a Cartesian product to generate a combined time queue. This combined time queue contains all possible combinations of precision offsets, with each combination of precision offsets forming a time unit. Perform time Pareto pruning on the combined time queue, selecting the Top-K time units at the Pareto front and using them as new time Pareto queues. Construct a parent node corresponding to the two child nodes, store the new time Pareto queue inside the parent node, and add the parent node to the previous level. Update the previous level to the current level. If the current level contains multiple nodes, continue iterating. When the current level contains only one node, terminate the iteration and set that node as the root node of the time search tree, and obtain the time Pareto queue of that root node.

[0013] Optionally, the quantization process of the pre-trained DiT model based on the static weight accuracy and the optimal scheduling path includes: During the inference process of the pre-trained DiT model, static weight precision is used as the fixed weight precision of the pre-trained DiT model; For each key time period, the precision offset corresponding to the key time period in the optimal scheduling path is obtained. The sum of the baseline activation precision and the corresponding precision offset is taken as the application activation precision for that key time period. The activation value of the pre-trained DiT model is quantized using the application activation precision to realize the quantization processing of the pre-trained DiT model.

[0014] According to a second aspect of this application, a spatiotemporal hybrid precision quantization system for diffusion models based on tree search is provided, comprising: The spatial search module is used to construct a spatial search tree and determine the static weight accuracy and baseline activation accuracy of the DiT model based on the spatial search tree. A time-segmentation model is used to discretize the time dimension of the DiT model to generate key time periods; based on the baseline activation accuracy, the average number of activation bits and cumulative distortion for each key time period are obtained. The time search module is used to construct a time search tree, and aggregate the time search tree using the average number of activation bits and the cumulative distortion to obtain the optimal scheduling path of the DiT model. The quantization module is used to quantize the pre-trained DiT model based on the static weight accuracy and the optimal scheduling path to obtain a quantized DiT model.

[0015] According to a third aspect of this application, an image generation method is provided, comprising: Determine the pre-trained FLUX text-to-image model; The pre-trained FLUX text-to-graph model is quantized using any of the tree-search-based diffusion model spatiotemporal hybrid precision quantization methods provided in the first aspect of this application to determine the quantized FLUX text-to-graph model. The preset image-generated text is input into the quantized FLUX text-generated image model to determine the generated image.

[0016] According to a fourth aspect of this application, a video generation method is provided, comprising: Determine the pre-trained Wan video generation model; The pre-trained Wan video generation model is quantized using any of the tree-search-based diffusion model spatiotemporal hybrid precision quantization methods provided in the first aspect of this application to determine the quantized Wan video generation model. The preset video generation text is input into the quantized Wan video generation model to determine the generated video.

[0017] This application provides a spatiotemporal hybrid precision quantization method for diffusion models based on tree-structured search. It constructs a temporal and spatial isomorphic search architecture using spatial and temporal search trees. The method aggregates the temporal search tree using the baseline activation precision of the spatial dimension, achieving a high degree of unification between the spatial and temporal dimensions of the DiT model. Segmented aggregation processing of the temporal dimension enables high-fidelity, low-latency spatiotemporal hybrid precision inference of the DiT model under extremely low average activation bit count budgets. The quantized DiT model of this application can be applied to scenarios involving the generation of image and video data. Through the highly unified logic of spatial and temporal dimensions, it can minimize data distortion under extremely low bit count budgets, making it suitable for ultra-high-definition video encoding, mobile device image editing (such as portrait generation and landscape optimization), and other scenarios without relying on cloud computing power.

[0018] Other technical effects resulting from the additional features will be further illustrated in the corresponding embodiments. Attached Figure Description

[0019] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart of a method for quantizing the spatiotemporal mixing accuracy of a diffusion model in one embodiment of this application; Figure 2 This is a flowchart of the spatiotemporal mixing accuracy quantization method for diffusion models in an application example of this application; Figure 3 This is a schematic diagram of temporal / spatial pruning search in an application example of this application; Figure 4 This is a schematic diagram of a diffusion model spatiotemporal mixing precision quantization system in one embodiment of this application. Detailed Implementation

[0020] The present application will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application, and these all fall within the protection scope of the present application. Parts not described in detail in the following embodiments can be implemented using existing technology.

[0021] It should be noted that all information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with relevant regulations.

[0022] Mixed-precision quantization is a key technique for reducing the inference cost of diffusion models. However, existing quantization search methods for diffusion models suffer from low spatial search efficiency and coarse temporal strategies, failing to fully explore the solution space and resulting in a low upper limit for mixed precision, thus failing to guarantee global optimum. To address these issues, this application provides a spatiotemporal mixed-precision quantization method for diffusion models based on tree search, thereby resolving these problems.

[0023] Reference Figure 1 and Figure 2 As shown, this application provides a method for spatiotemporal mixing precision quantization of diffusion models based on tree search, including: S1. Construct a spatial search tree, aggregate the spatial search tree, and obtain the static weight accuracy and baseline activation accuracy of the DiT model; the DiT model is used to perform image or video generation tasks. S2. Discretize the time dimension of the DiT model to generate key time periods; based on the baseline activation accuracy, obtain the average number of activation bits and cumulative distortion for each key time period; S3. Construct a time search tree, and aggregate the time search tree using the average number of activation bits and the cumulative distortion to obtain the optimal scheduling path for the DiT model. S4. Based on the static weight accuracy and the optimal scheduling path, the pre-trained DiT model is quantized to obtain the quantized DiT model. For example, this application proposes a dual tree-structured search method, which includes two cascaded stages: "spatial tree search" and "temporal tree search". This application is an efficient quantitative configuration search method for Diffusion Transformer (DiT) generative models, which adopts tree-structured Pareto pruning strategies in both spatial (network) and temporal (steps) dimensions.

[0024] The embodiments described above in this application construct a temporally and spatially isomorphic search architecture based on spatial search trees and temporal search trees. By aggregating the temporal search tree in the temporal dimension using the baseline activation precision of the spatial dimension, a high degree of unification between the spatial and temporal dimensions of the DiT model is achieved. Segmented aggregation processing of the temporal dimension enables high-fidelity, low-latency spatiotemporal hybrid precision inference of the DiT model under extremely low average activation bit count budgets. The quantized DiT model of this application can be applied to scenarios involving the generation of image, video, and other data. Through highly unified logic in the spatial and temporal dimensions, it can minimize data distortion under extremely low bit count budgets, making it suitable for ultra-high-definition video encoding, mobile device image generation (such as portrait generation and landscape generation), and other scenarios without relying on cloud computing power.

[0025] In some specific embodiments of this application, constructing a spatial search tree and aggregating the spatial search tree to obtain the static weight accuracy and baseline activation accuracy of the DiT model may further include: S11. Construct a candidate configuration set and a spatial search tree. The candidate configuration set includes several sets of quantized configurations. Each set of quantized configurations includes weight quantization precision and activation quantization precision. Each leaf node in the spatial search tree corresponds to a network layer in the DiT model. S12. For each leaf node in the spatial search tree, perform the following operations: traverse all quantization configurations in the candidate configuration set, calculate the average bit and quantization error of the leaf node under each quantization configuration, construct the spatial Pareto queue of the leaf node, the spatial Pareto queue contains several quantization units, each quantization unit includes the quantization configuration, its corresponding average bit and quantization error, and store the constructed spatial Pareto queue inside the corresponding leaf node. S13. Aggregate the spatial search tree to obtain the spatial Pareto queue of the root node of the spatial search tree. S14. Select the quantization configuration corresponding to the minimum average bit value and the minimum quantization error from the spatial Pareto queue of the spatial search tree root node, and use it as the benchmark quantization configuration; set the weight quantization precision in the benchmark quantization configuration to the static weight precision, and set the activation quantization precision in the benchmark quantization configuration to the benchmark activation precision.

[0026] In the above embodiments of this application, the spatial search tree is used to determine the static weight accuracy and the baseline activation accuracy, specifically including the following steps: Step 11: Treat the N network layers of the DiT model as linear sequence nodes in space to construct a spatial search tree. The N network layers of the DiT model correspond one-to-one with the N leaf nodes of the spatial search tree in topological order (i.e., forward propagation order). Step 12, Leaf Node Initialization: Calculate the leaf node corresponding to each network layer in the candidate configuration set. The initial spatial Pareto queue is constructed using the quantization error (MSE) and average bit depth. ; Step 13: Spatial tree aggregation to obtain a spatial configuration library; Step 14: Output the space allocation library: In the space search tree, the space Pareto queue of the root node is the global space allocation library. Select a baseline quantization configuration from the global space configuration library that meets the preset weight bit threshold. .

[0027] Locked weights: Benchmark quantization configuration Weight quantization accuracy The weights will remain unchanged in all subsequent time steps of the model (static weights).

[0028] Benchmark activation: Benchmark quantization configuration Activation quantization precision As a reference benchmark for model time search, the actual activation quantization accuracy of the DiT model is taken within a preset range of fluctuation, with the benchmark activation accuracy as the reference point.

[0029] In some specific embodiments of this application, the spatial search tree is aggregated to obtain a spatial Pareto queue of the root node of the spatial search tree; this may further include: Taking the level of the leaf node in the spatial search tree as the current level, iteratively perform the following operations: S131. Create a new parent level above the current level; S132. Traverse all nodes in the current level in topological order, select each pair of adjacent nodes as child nodes, combine the spatial Pareto queues stored in the selected child nodes by Cartesian product to generate a combined quantization queue. This combined quantization queue contains all possible quantization configuration combinations, and each quantization configuration combination constitutes a quantization unit. Perform spatial Pareto pruning on the combined quantization queue, select the Top-K quantization units at the Pareto front, and use them as a new spatial Pareto queue. Construct a parent node corresponding to the two child nodes, store the new spatial Pareto queue inside the parent node, and add the parent node to the previous level. S133. Update the previous level to the current level. If the current level contains multiple nodes, continue iterating. When the current level contains only one node, terminate the iteration and take that node as the root node of the spatial search tree, and obtain the spatial Pareto queue of that root node.

[0030] In the above embodiments of this application, during the spatial tree-like aggregation process, the linear topology of DiT is used to merge the spatial Pareto queues of adjacent nodes from bottom to top. The merging logic is as follows: (Cartesian product) in, A Pareto queue for storing one child node; A Pareto queue for storing space for another child node; For Cartesian product operations; The combined quantization queue contains all possible combinations of quantization configurations. Each combination of quantization configurations constitutes a quantization unit. Each quantization unit in the parent node's combined quantization queue consists of the following elements: Quantization configuration: A combination of quantization configurations for two child nodes; Average bit: The arithmetic mean of the average bit summation values ​​of the two child node quantization units; Quantization error: The sum of the quantization errors of the two child node quantization units; Spatial Pareto pruning: For the merged combined quantization queue, only the top-order quantization elements that are at the Pareto front in the two-dimensional space of "average bits - quantization error" are retained. A quantitative configuration.

[0031] It should be noted that during the traversal of nodes in the current level, every two adjacent nodes are selected from left to right (for example, the 1st and 2nd nodes, the 3rd and 4th nodes, and so on) to form a node pair to be processed. If the total number of nodes in the current level is odd, the rightmost unpaired node is directly added to the previous level as its own parent node, and its storage space is preserved in the Pareto queue.

[0032] In some specific embodiments of this application, the time dimension of the DiT model is discretized to generate key time periods; based on the baseline activation accuracy, the average number of activation bits and cumulative distortion for each key time period are obtained, which may further include: S21. Divide the total number of inference steps in the DiT model inference process into several non-overlapping key time periods; S22. For each critical time period, determine the actual activation accuracy for that critical time period based on the baseline activation accuracy. The expression for the actual activation accuracy is as follows: A real,j = A base + δ j δ j ∈Δ{-1,0,+1} Among them, A real,j A represents the actual activation precision during the j-th critical time period; base As the baseline activation accuracy; δ j Let be the precision offset for the j-th critical time period; Δ is the set of precision offsets. S23. For each critical time period, perform the following operations: traverse all precision offsets in the precision offset set, determine the corresponding actual activation precision based on the precision offset, and calculate the average number of activation bits and cumulative distortion when the DiT model performs inference with the actual activation precision during the critical time period.

[0033] In the embodiments described above, in order to transform continuous time steps into searchable nodes, the DiT model is discretized in the time dimension and subjected to sensitivity modeling, including: Step 21, Time Discretization: The total number of inference steps... (e.g., 50 steps) divided into The key time periods are denoted as follows: For example: every 5 steps constitutes a key time period, with a total of 10 key time periods. Each key time period shares the same set of precision offsets. Step 22, Time Sensitivity Assessment: For each key time period The sensitivity of the device to changes in activation accuracy was evaluated, including the action space and error measurement steps. Action space: Defined relative to the baseline activation precision Precision offset set ; Error metric: Calculate the applied precision offset δj during this critical time period. Then, the cumulative distortion of the generated results relative to full-precision inference. Specifically, feature map MSE accumulation calculation can be used: using the latent space feature map of the full-precision inference of the DiT model (δj=0) as the baseline feature map, the applied precision offset of each time step within the key time period is calculated. The cumulative distortion is obtained by summing the mean square error (MSE) between the post-feature map and the baseline feature map.

[0034] In some specific embodiments of this application, a time search tree is constructed, and the time search tree is aggregated using the average number of activation bits and the cumulative distortion to obtain the optimal scheduling path of the DiT model. This may further include: S31. Construct a time search tree, where each leaf node corresponds to a key time period; S32. For each leaf node in the time search tree, perform the following operations: construct a time Pareto queue for the leaf node. The time Pareto queue contains several time units. Each time unit includes a precision offset, its corresponding average number of active bits, and cumulative distortion. Store the constructed time Pareto queue inside the corresponding leaf node. S33. Aggregate the time search tree to obtain the time Pareto queue of the root node of the time search tree. S34. Select the precision offset corresponding to the time Pareto queue of the root node of the time search tree that has an average number of active bits less than the preset active bit threshold and the minimum cumulative distortion, and use it as the optimal scheduling path.

[0035] In the above embodiments of this application, the time search tree is used to determine the dynamic activation schedule, specifically including the following steps: Step 31: Utilize the sequence characteristics of key time periods ( Construct a time search tree, and the tree-like aggregation and pruning methods of the time search tree are consistent with those of the spatial search tree; Step 32, Initialize Time Leaf Nodes: Construct an initial time Pareto queue for the leaf nodes corresponding to each key time period. The elements of the time Pareto queue are ,in, The average number of active bits after applying the precision offset for this critical time period; The cumulative distortion introduced during this critical time period; Step 33: Time-tree aggregation to obtain a set of globally optimal scheduling schemes; Step 34, Root Node Decision: After After this merging, the root node of the time search tree contains a set of globally optimal scheduling schemes covering the entire time period, based on the user-defined preset activation bit threshold. Select the corresponding optimal scheduling path ,in, This represents the optimal precision offset for the j-th critical time period.

[0036] In some specific embodiments of this application, aggregating the time search tree to obtain the time Pareto queue of the root node of the time search tree may further include: Using the level of the leaf node in the time search tree as the current level, iteratively perform the following operations: S331. Create a new parent level above the current level; S332. Traverse all nodes in the current level in topological order, select each pair of adjacent nodes as child nodes, combine the time Pareto queues stored in the selected two child nodes by Cartesian product to generate a combined time queue. This combined time queue contains all possible combinations of precision offsets, and each combination of precision offsets constitutes a time unit. Perform time Pareto pruning on the combined time queue, select the Top-K time units at the Pareto front, and use them as new time Pareto queues. Construct a parent node corresponding to the two child nodes, store the new time Pareto queue inside the parent node, and add the parent node to the previous level. S333. Update the previous level to the current level. If the current level contains multiple nodes, continue iterating. When the current level contains only one node, terminate the iteration and take that node as the root node of the time search tree, and obtain the time Pareto queue of that root node.

[0037] For example, during the time-tree aggregation process, adjacent key time periods are merged pairwise in chronological order, such as merging the first key time period. and the second key time period ; Traverse the first key time interval corresponding to the first Pareto queue. The first critical time period corresponds to the second Pareto queue. All candidate actions (i.e., precision offsets) are combined.

[0038] Each precision offset combination constitutes a time unit, and each time unit in the parent node's combined time queue consists of the following elements: Precision offset: The combination of the precision offsets of the two child nodes; Average number of active bits: the arithmetic mean of the cumulative average number of active bits in the two child node time units, i.e. ,in, The average number of active bits in the parent node. This represents the average number of active bits in one of the child nodes corresponding to the parent node. This represents the average number of active bits in the corresponding child node of the parent node. Cumulative distortion: After applying the precision offset of the corresponding time unit to each of the two child nodes, the model is forward-propagated for the time steps contained in the two time units. The mean square error (MSE) of the output of this propagation process is then calculated between the output of the full-precision model and the output of the full-precision model. The result is the cumulative distortion of the parent node. The full-precision model refers to the baseline model that uses general-purpose 32-bit single-precision floating-point numbers (FP32) to complete all calculations, data storage, and forward propagation.

[0039] Time Pareto pruning: After each merge, the resulting combined time queue is forcefully pruned as follows: Sort by average number of activation bits Sort; Filtering: Eliminate all dominated solutions. The definition of a dominated solution is as follows: In the pruning process of the combined time queue, if there exists a candidate solution X (i.e., time unit) such that the average number of activation bits of another candidate solution Y is lower than that of candidate solution X and the cumulative distortion is smaller than that of candidate solution X, then candidate solution X is a dominated solution and should be directly eliminated.

[0040] Truncation: Retain Top- The optimal path segment (i.e., time unit).

[0041] The embodiments described above employ a time-dimensional tree-like aggregation and pruning strategy, which can plan resource allocation throughout the entire generation process from a global perspective (such as automatically allocating high precision during critical structure generation periods and low precision during non-critical periods). This can minimize generation distortion while strictly satisfying the average bit count constraint, and solves the technical problem that existing time step allocation methods often use static quantization or local greedy strategies, which cannot perceive redundancy differences throughout the entire process, resulting in wasted computing power in non-critical stages or insufficient precision in critical stages.

[0042] In some specific embodiments of this application, quantization processing of the pre-trained DiT model based on static weight accuracy and optimal scheduling path may further include: S41. During the inference process of the pre-trained DiT model, static weight precision is used as the fixed weight precision of the pre-trained DiT model. S42. For each key time period, obtain the precision offset corresponding to the key time period in the optimal scheduling path, and take the sum of the baseline activation precision and the corresponding precision offset as the application activation precision for that key time period. Use the application activation precision to quantize the activation value of the pre-trained DiT model to achieve the quantization processing of the pre-trained DiT model.

[0043] For example, the quantized DiT model deployment employs a static weight, segmented dynamic activation mode, including: Step 41, Weight Loading: Load the baseline quantization configuration. Determined fixed-precision weights.

[0044] Step 42, Inference Control: Set a time period counter; when inference enters a critical time period... At that time, read the corresponding optimal precision offset. The activation precision of each network layer at the current moment = Finally, perform inference calculations.

[0045] The embodiments described above in this application propose a spatiotemporally isomorphic dual tree-structured search framework, which has the following advantages: 1. Spatiotemporal algorithm unification: This application innovatively proposes to regard the "time step sequence" as a linear structure isomorphic to the "network layer sequence", and also adopts tree-like Pareto aggregation and pruning strategies in the time dimension to achieve a high degree of unification of algorithm logic.

[0046] 2. Global Pareto Optimality: By constructing Pareto fronts in both time and space dimensions, the final generated time-space curvature scheme is ensured to achieve theoretically minimized distortion while strictly satisfying hardware constraints (such as the average number of bits).

[0047] 3. Deployment friendliness: Adopting a strategy of "static weights and dynamic activations" combined with time-dimension segmented aggregation, it not only utilizes time redundancy but also avoids the IO overhead caused by frequent weight switching.

[0048] Specifically, this application adopts a spatiotemporal isomorphic search architecture with dual tree-like Pareto aggregation, which treats the time step sequence as a linear structure isomorphic to the network layer sequence for unified modeling. This can achieve a high degree of unification of spatial (hierarchical) and temporal (step) search logic and Pareto optimality guarantee. It solves the technical problem in the prior art where spatial and temporal search algorithms are separated (such as using genetic algorithms and dynamic programming respectively), resulting in high system complexity, difficulty in hyperparameter tuning, and inability to guarantee spatiotemporal joint global optimality.

[0049] The embodiments described above employ a linear search algorithm based on Top-K pruning and a deployment strategy of "static weights + segmented dynamic activation," which can achieve efficient strategy search at the minute level. This solves the technical problems of traditional automated search methods, such as exponential space explosion leading to extremely high computational costs, and the increased inference latency caused by memory bandwidth limitations in actual hardware deployments due to dynamic weight switching.

[0050] In summary, this application combines the aforementioned spatiotemporal isomorphic search architecture, global time resource planning, and efficient pruning deployment strategy. This enables high-fidelity, low-latency spatiotemporal mixed-precision inference of the diffusion model under extremely low average bit budget, overcoming the overall technical problems of low search efficiency, insufficient utilization of temporal redundancy, and difficulty in engineering the implementation of mixed-precision solutions in existing technologies. Addressing the issues of shakiness, blurriness, and limited shooting angles often found in short video / image footage shot by ordinary users, the quantized model provided in this application can directly generate high-definition, clear short videos / images (such as sharp landscapes, natural portraits, and food images with distinct textures) from original low-quality footage, within the constraints of mobile phone computing power and battery power consumption. This avoids overheating and lag caused by excessive resource consumption during the generation process, ensuring that ideal finished images are generated instantly after shooting, without the need for complex post-processing editing.

[0051] The following examples will be used to further illustrate this application in order to better understand the above-mentioned technical solutions. It should be understood that the following are only some examples and are not intended to limit this application.

[0052] Application Example 1: W4A4 spatiotemporal mixing precision configuration of the DiT-XL / 2 model, including the following steps: Step 1: Construct a spatial search tree TTS and perform a spatial search: Running a space search tree, setting the weight target to 4 bits, yields the baseline quantization configuration. The weight quantization precision is 4-bit, and the activation quantization precision is 4-bit on average.

[0053] Step 2, Time Discretization: Divide the 50 inference steps into 10 time periods (5 steps per period).

[0054] Step 3, Time-based Tree Search: Step 31: Construct the leaf nodes of the time search tree: Calculate the precision offset for each key time period. The distortion metric MSE under offset was found to be most sensitive in the critical time period 3-6 (the middle segment); Step 32: Perform aggregation pruning: Merge adjacent nodes in the time search tree sequentially to obtain a local optimum; prune in Pareto space; repeat merging of nodes to the root node; Pruning results: Under the constraint of an average target activation of 4 bits, the algorithm automatically selected the following strategy: (At the beginning, i.e., the first and second critical time periods): Precision offset is -1 (speed increase). (In the middle, i.e., the third to sixth critical time periods): The precision offset is +0 or +1 (quality guaranteed). (End, i.e., the seventh to tenth critical time periods): Precision offset is -1 (speed increase) Reference Figure 3 The diagram shown illustrates the principle of aggregation pruning in a spatial / temporal search tree, where P... i For the i-th time / space Pareto queue, the configuration includes the quantization configuration, the corresponding average bits and quantization error for the spatial Pareto queue, and the precision offset, the corresponding average number of active bits and cumulative distortion for the time Pareto queue.

[0055] This application example, while maintaining an average activation precision of 4-bit, significantly outperforms the FID (Fixed Detection Index) scheme with a fixed 4-bit precision throughout the entire process. Furthermore, compared to a simple linear decay strategy, it better captures the model's specific sensitivity in the intermediate stages (i.e., the distortion metric MSE). This DiT-XL / 2 model can be further applied to image generation. Because the model maintains an average activation precision of 4-bit, it can significantly reduce the computational and memory overhead of cloud rendering, supporting the parallel generation of several ultra-high-definition images (such as batch generation of e-commerce product detail images). The generated ultra-high-definition images exhibit almost indistinguishable details from the actual photographs, avoiding texture blurring and color distortion caused by linear decay. This allows the generated images to meet both the requirements of lightweight and rapid generation and accurate reproduction of the actual objects.

[0056] This application provides a spatiotemporal hybrid precision quantization method for diffusion models based on tree search, which can be applied to text-generated image models and text-generated video models.

[0057] In some specific embodiments of this application, a spatiotemporal mixing precision quantization method for diffusion models based on tree search can be applied to FLUX text-generated image models. Specifically, an image generation method includes: Step 1: Determine the pre-trained FLUX text-to-image model; Step 2: Use the spatiotemporal hybrid precision quantization method of diffusion model based on tree search in any of the above embodiments to quantize the pre-trained FLUX text graph model and determine the quantized FLUX text graph model. Step 3: Input the preset image-generated text into the quantized FLUX text-generated image model to determine the generated image.

[0058] The image generation method of the above embodiments of this application further reduces the number of quantization bits, reduces quantization error and distortion, and improves the image generation quality by quantizing the FLUX text-to-image model.

[0059] In some specific embodiments of this application, a spatiotemporal mixing precision quantization method for diffusion models based on tree search can be applied to Wan video generation models. Specifically, a video generation method includes: Step 1: Determine the pre-trained Wan video generation model; Step 2: Use the spatiotemporal hybrid precision quantization method of diffusion model based on tree search in any of the above embodiments to quantize the pre-trained Wan video generation model and determine the quantized Wan video generation model. Step 3: Input the preset video generation text into the quantized Wan video generation model to determine the generated video.

[0060] The image generation method of the above embodiments of this application further reduces the number of quantization bits, reduces quantization error and distortion, and improves video generation quality by quantizing the Wan video generation model.

[0061] Based on the same inventive concept, another embodiment of this application provides a spatiotemporal hybrid precision quantization system for diffusion models based on tree search, referring to... Figure 4 As shown, the diffusion model spatiotemporal mixing precision quantization system 100 includes: The spatial search module 110 is used to construct a spatial search tree and determine the static weight accuracy and baseline activation accuracy of the DiT model based on the spatial search tree. The time segmentation model 120 is used to discretize the time dimension of the DiT model to generate key time periods; based on the baseline activation accuracy, the average number of activation bits and cumulative distortion of each key time period are obtained. The time search module 130 is used to construct a time search tree, and aggregate the time search tree using the average number of activation bits and the cumulative distortion to obtain the optimal scheduling path of the DiT model. The quantization module 140 is used to quantize the pre-trained DiT model based on the static weight accuracy and the optimal scheduling path to obtain the quantized DiT model.

[0062] It should be noted that the modules in the diffusion model spatiotemporal mixing precision quantization system based on tree search provided in the above embodiments of this application correspond to the steps of the diffusion model spatiotemporal mixing precision quantization method based on tree search in any of the above embodiments. Those skilled in the art can refer to the step features of the diffusion model spatiotemporal mixing precision quantization method based on tree search to implement the corresponding modules in the diffusion model spatiotemporal mixing precision quantization system based on tree search, which will not be elaborated here.

[0063] In another embodiment of this application, an electronic device is also provided, including a memory and a processor; the memory is used to store program instructions; the processor is used to call the program instructions stored in the memory and execute the steps of the above-described diffusion model spatiotemporal mixing precision quantization method based on tree search according to the obtained program instructions.

[0064] Optionally, the memory is used to store programs; the memory may include volatile memory, such as random-access memory (RAM), such as static random-access memory (SRAM), double data rate synchronous dynamic random-access memory (DDR SDRAM), etc.; the memory may also include non-volatile memory, such as flash memory. The memory is used to store computer programs (such as application programs and functional modules that implement the above methods), computer instructions, etc., and the aforementioned computer programs and computer instructions can be partitioned and stored in one or more memories. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by the processor.

[0065] The aforementioned computer programs, computer instructions, etc., can be stored in partitions within one or more memory locations. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by a processor.

[0066] A processor is used to execute a computer program stored in memory to implement the various steps of the methods involved in the above embodiments. For details, please refer to the relevant descriptions in the preceding method embodiments.

[0067] The processor and memory can be separate structures or integrated structures. When the processor and memory are separate structures, they can be coupled together via a bus.

[0068] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0069] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0070] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0071] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0072] The preferred features in the above embodiments can be used individually in any embodiment, or in any combination thereof, provided they do not conflict with each other. Furthermore, parts not described in detail in the embodiments can be implemented using existing technologies.

[0073] The foregoing has described some specific embodiments of this application. It should be understood that this application is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the substantive content of this application. The above-described preferred features can be used in any combination without conflict.

Claims

1. A spatiotemporal hybrid precision quantization method for diffusion models based on tree search, characterized in that, include: Construct a spatial search tree, aggregate the spatial search tree, and obtain the static weight accuracy and baseline activation accuracy of the DiT model; The DiT model is used to perform image or video generation tasks; The time dimension of the DiT model is discretized to generate key time periods; based on the baseline activation accuracy, the average number of activation bits and cumulative distortion for each key time period are obtained. Construct a time search tree, and aggregate the time search tree using the average number of activation bits and the cumulative distortion to obtain the optimal scheduling path for the DiT model; The pre-trained DiT model is quantized based on the static weight accuracy and optimal scheduling path to obtain the quantized DiT model.

2. The spatiotemporal mixing accuracy quantization method for diffusion models based on tree search according to claim 1, characterized in that, The construction of the spatial search tree, and the aggregation of the spatial search tree to obtain the static weight accuracy and baseline activation accuracy of the DiT model, includes: Construct a candidate configuration set and a spatial search tree. The candidate configuration set includes several groups of quantized configurations. Each group of quantized configurations includes weight quantization precision and activation quantization precision. Each leaf node of the spatial search tree corresponds to a network layer in the DiT model. For each leaf node in the spatial search tree, the following operations are performed: traverse all quantization configurations in the candidate configuration set, calculate the average bit and quantization error of the leaf node under each quantization configuration, construct the spatial Pareto queue of the leaf node, the spatial Pareto queue contains several quantization units, each quantization unit includes the quantization configuration and its corresponding average bit and quantization error, and store the constructed spatial Pareto queue inside the corresponding leaf node. Aggregate the spatial search tree to obtain the spatial Pareto queue of the root node of the spatial search tree; The quantization configuration corresponding to the minimum quantization error when the average bit value is less than the preset weight bit threshold is selected from the spatial Pareto queue of the root node of the spatial search tree, and is used as the benchmark quantization configuration; the weight quantization precision in the benchmark quantization configuration is set to the static weight precision, and the activation quantization precision in the benchmark quantization configuration is set to the benchmark activation precision.

3. The spatiotemporal mixing accuracy quantization method for diffusion models based on tree search according to claim 2, characterized in that, The aggregation of the spatial search tree to obtain the spatial Pareto queue of the root node of the spatial search tree includes: Taking the level of the leaf node in the spatial search tree as the current level, iteratively perform the following operations: Create a new parent level above the current level; Traverse all nodes in the current level in topological order, select each pair of adjacent nodes as child nodes, combine the spatial Pareto queues stored in the selected child nodes by Cartesian product to generate a combined quantization queue. This combined quantization queue contains all possible quantization configuration combinations, and each quantization configuration combination constitutes a quantization unit. Perform spatial Pareto pruning on the combined quantization queue, select the Top-K quantization units at the Pareto front, and use them as a new spatial Pareto queue. Construct a parent node corresponding to the two child nodes, store the new spatial Pareto queue inside the parent node, and add the parent node to the previous level. Update the previous level to the current level. If the current level contains multiple nodes, continue iterating. When the current level contains only one node, terminate the iteration and set that node as the root node of the spatial search tree, and obtain the spatial Pareto queue of that root node.

4. The spatiotemporal mixing accuracy quantization method for diffusion models based on tree search according to claim 1, characterized in that, The time dimension of the DiT model is discretized to generate key time periods; Based on the aforementioned baseline activation accuracy, the average number of activated bits and cumulative distortion for each key time period are obtained, including: The total number of inference steps in the DiT model inference process is divided into several non-overlapping key time periods; For each critical time period, the actual activation accuracy for that critical time period is determined based on the baseline activation accuracy. The expression for the actual activation accuracy is as follows: A real,j = A base + d j ,d j ∈Δ{-1,0,+1} Among them, A real,j A represents the actual activation precision during the j-th critical time period; base As the baseline activation accuracy; δ j Let be the precision offset for the j-th critical time period; Δ is the set of precision offsets. For each critical time period, the following operations are performed: traverse all precision offsets in the precision offset set, determine the corresponding actual activation precision based on the precision offset, and calculate the average number of activation bits and cumulative distortion when the DiT model performs inference with the actual activation precision during the critical time period.

5. The spatiotemporal mixing accuracy quantization method for diffusion models based on tree search according to claim 1, characterized in that, The construction of the time search tree, which aggregates the time search tree using the average number of activation bits and the cumulative distortion, yields the optimal scheduling path for the DiT model, including: Construct a time search tree, wherein each leaf node of the time search tree corresponds to a key time period; For each leaf node in the time search tree, perform the following operations: construct a time Pareto queue for that leaf node, the time Pareto queue containing several time units, each time unit including a precision offset, its corresponding average number of active bits and cumulative distortion, and store the constructed time Pareto queue inside the corresponding leaf node; Aggregate the time search tree to obtain the time Pareto queue of the root node of the time search tree; The optimal scheduling path is selected from the time Pareto queue of the root node of the time search tree, based on the precision offset corresponding to the condition that the average number of active bits is less than the preset active bit threshold and the cumulative distortion is minimized.

6. The spatiotemporal mixing accuracy quantization method for diffusion models based on tree search according to claim 5, characterized in that, Aggregating the time search tree yields the time Pareto queue of the root node, including: Using the level of the leaf node in the time search tree as the current level, iteratively perform the following operations: Create a new parent level above the current level; Traverse all nodes in the current level in topological order, selecting each pair of adjacent nodes as child nodes. Combine the time Pareto queues stored in the selected child nodes using a Cartesian product to generate a combined time queue. This combined time queue contains all possible combinations of precision offsets, with each combination of precision offsets forming a time unit. Perform time Pareto pruning on the combined time queue, selecting the Top-K time units at the Pareto front and using them as new time Pareto queues. Construct a parent node corresponding to the two child nodes, store the new time Pareto queue inside the parent node, and add the parent node to the previous level. Update the previous level to the current level. If the current level contains multiple nodes, continue iterating. When the current level contains only one node, terminate the iteration and set that node as the root node of the time search tree, and obtain the time Pareto queue of that root node.

7. The spatiotemporal mixing accuracy quantization method for diffusion models based on tree search according to claim 5, characterized in that, The quantization process of the pre-trained DiT model based on the static weight accuracy and optimal scheduling path includes: During the inference process of the pre-trained DiT model, static weight precision is used as the fixed weight precision of the pre-trained DiT model; For each key time period, the precision offset corresponding to the key time period in the optimal scheduling path is obtained. The sum of the baseline activation precision and the corresponding precision offset is taken as the application activation precision for that key time period. The activation value of the pre-trained DiT model is quantized using the application activation precision to realize the quantization processing of the pre-trained DiT model.

8. A spatiotemporal hybrid precision quantization system for diffusion models based on tree search, characterized in that, include: The spatial search module is used to construct a spatial search tree and determine the static weight accuracy and baseline activation accuracy of the DiT model based on the spatial search tree. A time-segmentation model is used to discretize the time dimension of the DiT model to generate key time periods; based on the baseline activation accuracy, the average number of activation bits and cumulative distortion for each key time period are obtained. The time search module is used to construct a time search tree, and aggregate the time search tree using the average number of activation bits and the cumulative distortion to obtain the optimal scheduling path of the DiT model. The quantization module is used to quantize the pre-trained DiT model based on the static weight accuracy and the optimal scheduling path to obtain a quantized DiT model.

9. An image generation method, characterized in that, include: Determine the pre-trained FLUX text-to-image model; The pre-trained FLUX text-to-graph model is quantized using any one of the tree-search-based diffusion model spatiotemporal hybrid precision quantization methods described in claims 1 to 7, thereby determining the quantized FLUX text-to-graph model. The preset image-generated text is input into the quantized FLUX text-generated image model to determine the generated image.

10. A video generation method, characterized in that, include: Determine the pre-trained Wan video generation model; The pre-trained Wan video generation model is quantized using any one of the tree-search-based diffusion model spatiotemporal hybrid precision quantization methods described in claims 1 to 7 to determine the quantized Wan video generation model. The preset video generation text is input into the quantized Wan video generation model to determine the generated video.

Images