Time-sensitive diffusion model weight calibration method and content generation method

Abstract

This application provides a time-sensitive diffusion model weight calibration method and content generation method, including: calculating the sensitivity index of each layer of the diffusion model at different time steps using several calibration data; normalizing the sensitivity of all time steps to obtain a normalized time-series importance score; modeling the weight quantization problem as a weighted least squares problem based on the normalized time-series importance score; and solving the weighted least squares problem to obtain the calibrated quantized weights. This application employs time-series importance assessment and normalization techniques, enabling accurate positioning and weighting of key stages in the generated trajectory, solving the problem in traditional mean calibration methods where the fitting requirements of key time steps are diluted by non-key time steps. The quantized diffusion model calibrated using this method, when deployed on resource-constrained devices, can ensure the generation of semantically accurate and structurally stable virtual scenes at key stages, effectively improving the reliability and real-time performance of the device.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and lightweight large models, specifically to a time-sensitive diffusion model weight calibration method and content generation method. Background Technology

[0002] In applications that utilize large deployed models for content generation, especially in resource-constrained smart devices (such as in-vehicle terminals and mobile AR devices), existing technologies struggle to maintain sufficient semantic consistency and structural fidelity during critical generation stages. This can lead to issues such as misaligned guiding elements, distorted scene layouts, or failures in virtual-real fusion, which in turn affect the reliability and user experience of downstream applications such as augmented reality navigation and human-computer interaction.

[0003] In the field of lightweighting large models, weight quantization is a core technique for reducing model storage footprint and GPU bandwidth requirements. For diffusion models, the weight parameters are static during inference, but the input activations they process change drastically over time (evolving from Gaussian noise to a clear image). This contradiction between "static weights and dynamic inputs" presents a significant challenge to quantization calibration. Existing weight calibration techniques have the following shortcomings: The failure of the independent and identically distributed (i.i.d.) assumption: Mainstream weight quantization algorithms (such as GPTQ and AWQ) were originally designed for Large Language Models (LLMs). When constructing optimization objectives (such as minimizing reconstruction error), they assume by default that all samples in the calibration dataset are independent and identically distributed (i.i.d.) and equally important. However, in diffusion models, the distribution of input data at different time steps varies greatly (Distribution Shift), and their contributions to the final generation quality are completely different. Simply averaging the errors across all time steps will cause the fitting ability of the weights in critical generation stages (such as the high-noise graphing period) to be diluted by a large amount of data from non-critical stages.

[0004] Lack of trajectory-aware calibration mechanisms: Existing data selection methods (such as SalientData Selection in S2Q-VDiT) attempt to select "important" calibration samples, but this is a hard "keep vs. discard" selection that cannot achieve refined soft weighting. They mainly focus on the spatial dimension of token importance, but fail to incorporate the temporal dimension of trajectory evolution into the optimization objective of the weights.

[0005] Risk of structural collapse at low bit depths: At extremely low bit depths (e.g., 3-bit / 4-bit quantization), the expressive power of weights is significantly limited. If the calibration process is not tilted towards critical time steps, quantization noise in static weights can easily destroy the global structural information in the early stages of generation, leading to artifacts, blurring, or semantic errors in the generated image. Existing Min-Max or standard MSE optimization methods cannot effectively address this problem.

[0006] A search revealed a Chinese patent with patent application number 202510334094.4, which discloses a mixed-precision quantization method for a diffusion model based on dynamic sensitivity. Under given resource constraints, this method minimizes the sum of the sensitivities of each layer at all time steps to obtain an adaptive bit-width configuration strategy and a mixed-precision quantization diffusion model. This method can dynamically adjust the bit-width allocation of network layers according to the quantization sensitivity of the diffusion model at the denoising time step, thus better adapting to sensitivity changes during the denoising process. However, it does not consider incorporating the trajectory evolution over time into the weight optimization objective, leaving room for reduction in computational complexity.

[0007] Therefore, the existing weight calibration method with its limitations is used to quantize the diffusion model. The resulting low-bit model still cannot solve the above problems when deployed on resource-constrained smart devices, which in turn affects downstream applications such as augmented reality navigation and human-computer interaction. Summary of the Invention

[0008] To address one of the shortcomings of existing technologies, the purpose of this application is to provide a time-sensitive diffusion model weight calibration method and content generation method.

[0009] A first aspect of this application provides a weight calibration method based on a time-sensitive diffusion model, comprising: Using several calibration data, the sensitivity indices of each layer of the diffusion model at different time steps were calculated; The sensitivity of the entire time step is normalized to obtain a normalized temporal importance score; Based on the normalized temporal importance score, the weight quantization problem is modeled as a weighted least squares problem; The weighted least squares problem is solved to obtain the calibrated quantization weights.

[0010] Optionally, the sensitivity metric may include Fisher information or gradient norm.

[0011] Optionally, the weighted least squares problem specifically refers to: in The original full-precision weights for the diffusion model. To quantify weights, For time step Input features, This represents the normalized temporal importance score.

[0012] Optionally, solving the weighted least squares problem to obtain the quantized weights includes: Construct a time-weighted Hessian matrix; Using the Cholesky decomposition and inverse matrix solution method, based on the time-series weighted Hessian matrix, the optimal quantization weights and corresponding quantization parameters are solved for the weighted least squares problem.

[0013] Optionally, the construction of the time-weighted Hessian matrix includes: Calibration input data Multiply by the square root of the time importance fraction , obtain weighted input ; Construct the Hessian matrix based on the weighted input: Where T represents the total number of time steps in the diffusion model.

[0014] Optionally, the step of using Cholesky decomposition and inverse matrix solving to solve the weighted least squares problem for the time-series weighted Hessian matrix to obtain the optimal quantization weights and corresponding quantization parameters includes: Add damping terms to the diagonal elements of the time-weighted Hessian matrix to ensure its positive definiteness; The positive definite time-weighted Hessian matrix is ​​subjected to Cholesky decomposition to obtain the lower triangular matrix; Using the aforementioned lower triangular matrix, the weighted Hessian inverse matrix is ​​solved by substitution. The weighted Hessian inverse matrix and the full-precision weight matrix are processed in column order: After quantization of the j-th column of the full-precision weight matrix, the compensation update amount to be applied to each subsequent column is calculated based on the information of the corresponding column in the weighted Hessian inverse matrix, and the compensation update amount is applied to offset the quantization error of the current column; then the remaining submatrix is ​​updated, and the next column iteration is started. The column-by-column iterative process is executed repeatedly until all columns have been processed. The resulting quantization weight matrix has the minimum weighted reconstruction error at the key time step and outputs the quantization parameters adapted to the weights simultaneously.

[0015] A second aspect of this application provides a method for generating virtual content, comprising: Obtain virtual content generation instructions, the content of which describes information related to the virtual content to be generated; The diffusion model deployed on the smart device is invoked, and corresponding virtual content is generated based on the instructions, wherein the diffusion model is calibrated by any of the time-sensitive diffusion model weight calibration methods described above.

[0016] Optionally, the virtual content generation is a painting generation method on a mobile terminal, including: Receive drawing instructions input by the user, the drawing instructions including text descriptions or sketch information; The diffusion model deployed on the mobile terminal is invoked to generate image data based on the drawing instructions, wherein the diffusion model is calibrated by any of the time-sensitive diffusion model weight calibration methods described above.

[0017] Optionally, the virtual content generation is a virtual scene generation for the in-vehicle system, including: Receive scene generation instructions, which include natural language voice instructions, driver's gaze or gesture recognition results, current navigation path information or vehicle surrounding environment perception data; The diffusion model deployed on the vehicle terminal is invoked to generate virtual scene data in real time based on the scene generation instructions, which is used for augmented reality navigation, cockpit entertainment or human-computer interaction interface rendering. The diffusion model weight calibration method described in any one of the diffusion models is used for calibration.

[0018] A third aspect of this application provides a terminal including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, can perform any of the methods described above.

[0019] This application presents a time-sensitive diffusion model weight calibration method that employs time-importance assessment and normalization techniques to accurately locate and weight key stages in the generated trajectory. This addresses the problem in traditional mean calibration methods where the fitting requirements of key time steps are diluted by non-key time steps. The quantized diffusion model calibrated using this method, when deployed on resource-constrained devices, can guarantee the generation of semantically accurate and structurally stable virtual scenes at key stages, effectively improving the reliability and real-time performance of the device.

[0020] Other technical effects resulting from the additional features will be further illustrated in the corresponding embodiments. Attached Figure Description

[0021] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart illustrating a time-sensitive diffusion model weight calibration method according to an exemplary embodiment; Detailed Implementation The present application will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application, and these all fall within the protection scope of the present application. Parts not described in detail in the following embodiments can be implemented using existing technology.

[0022] The terms "comprising" and "having," and any variations thereof, in the embodiments of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such processes, methods, products, or devices.

[0023] Traditional mean calibration methods suffer from a dilution of the fitting requirements at critical time steps by non-critical time steps. To address this issue, this application provides a time-sensitive diffusion model weight calibration method to resolve these problems.

[0024] Reference Figure 1 In one embodiment, a time-sensitive diffusion model weight calibration method includes the following steps: S1, using several calibration data, calculate the sensitivity index of each layer of the diffusion model at different time steps; S2, normalize the sensitivity of the entire time step to obtain the normalized time series importance score; S3, based on normalized temporal importance scores, models the weight quantization problem as a weighted least squares problem; S4 solves the weighted least squares problem to obtain the calibrated quantization weights.

[0025] Specifically, the form of calibration data depends on the input modality of the diffusion model: for diffusion models that take images as input (such as those used for image inpainting or image-to-image generation), calibration data can be unlabeled real or synthetic images; for text-to-image diffusion models that take text as input (such as Flux.1-dev), calibration data is a set of text prompts oriented towards the target task; if the model supports multimodal input (such as text + perceptual data), the calibration data can contain corresponding combinations of multimodal samples.

[0026] Specifically, the calibrated quantization weights can be used to deploy compressed diffusion models, enabling them to maintain generation quality and structural stability even under low-bit (e.g., 3-bit or 4-bit) quantization conditions, significantly reducing model memory usage and computational overhead. This quantization model can run efficiently on resource-constrained edge devices, including mobile terminals and in-vehicle systems, without relying on cloud computing power, thus supporting real-time, low-latency, high-quality virtual content generation.

[0027] The above embodiments propose a time-sensitivity-based weighting mechanism, which enables static weights to "sense" which time steps are more critical during the calibration phase, thereby prioritizing the fitting of data distributions in these highly sensitive phases. This achieves accurate positioning and weighting of key phases (such as the mapping phase) in the generated trajectory, solving the problem that the fitting requirements of key time steps are diluted by non-key time steps in traditional mean calibration methods.

[0028] To obtain accurate sensitivity indices, in some specific embodiments of this application, step S1, which uses several calibration data to calculate the sensitivity indices of each layer of the diffusion model at different time steps t, can be achieved by the following steps: S11, Select several calibration samples, input them into the pre-trained diffusion model, perform forward propagation, and collect the activation data and model output data of each network layer at different denoising time steps t; S12, Based on the collected results, calculate the sensitivity index for each time step t; The sensitivity index is the diagonal approximation of Fisher information or the norm of the weight gradient (such as the L2 norm), which is used to reflect the criticality of the time step in generating the trajectory.

[0029] Specifically, the diagonal approximation of Fisher information refers to the calculation of the main diagonal elements of the outer product matrix of the gradient of the loss function with respect to the model weights at time step t.

[0030] Because the generation process of the diffusion model has significant temporal non-uniformity, the early composition stage is usually more sensitive to parameter errors than the later texture enhancement stage, directly determining whether the image structure collapses. Using Fisher scores as weights essentially leverages their characteristic of measuring "parameter sensitivity" to introduce a "value-oriented" approach to the quantization calibration of static weights: it forces the optimization algorithm to no longer treat all calibration data equally, but instead prioritizes limited precision resources to the key time steps that have the greatest impact on the final image quality, thereby ensuring that even at extremely low bit depths, the model can still preserve the generated "skeleton" and semantics, and not have the optimization objective diluted by a large amount of non-critical low-noise data.

[0031] Specifically, the norm of the weight gradient is defined as follows: at time step t, the gradient of the loss function is calculated for the weights of the l-th layer of the diffusion model, and the square of the Frobenius norm is taken.

[0032] The larger this value, the more sensitive the model output is to weight perturbations at that time step, and the more critical its impact on the final generation quality.

[0033] This metric directly reflects "how much output error is caused by a small change in parameters". During the diffusion generation process, the gradient magnitude is significantly higher in the composition stage (such as the mid-stage of denoising) than in the noise or detail stage, thus it can naturally identify the key time steps that play a decisive role in the image structure. At the same time, it has engineering advantages such as high computational efficiency, numerical stability, and no need for second derivatives, making it a lightweight and effective alternative to Fisher information.

[0034] The above embodiments achieve accurate positioning of key stages in the generated trajectory by calculating Fisher information or gradient norm as a time-series sensitivity index.

[0035] Furthermore, in some specific implementations, step S2 involves normalizing the sensitivity of the entire time step to obtain a normalized temporal importance score. Specifically: Normalization using softmax: in This is a hyperparameter that controls the distribution; the default value of 1 is sufficient. The importance score after normalization. For sensitivity.

[0036] Due to the different levels within the same layer The distribution varies greatly, and using simple summation and normalization would cause most time steps to be ignored.

[0037] In the above embodiments, a normalized temporal importance score is obtained by normalizing the sensitivity index of each time step. This operation not only eliminates the scale differences between different layers and time steps, but also enables key generation stages (such as the mapping period) to obtain higher fitting weights in the optimization objective.

[0038] Traditional post-training quantization methods (such as GPTQ) minimize the average reconstruction error, resulting in all time steps being treated equally. In some specific embodiments of this application, the weighted reconstruction error is minimized: in These are the original full-precision weights for the diffusion model. For the weights to be quantified, For time step Input activation value, Normalized temporal importance score.

[0039] The aforementioned weighted least squares problem is solved by introducing a temporal weighting mechanism, which prioritizes the protection of critical generation stages during the calibration process, significantly improving the structural stability and semantic consistency of the model under low bit conditions, and effectively alleviating the problem of critical information dilution caused by uniform fitting.

[0040] In order to incorporate the trajectory evolution over time into the optimization objective of the weights, in some specific embodiments of this application, S4, solving the weighted least squares problem to obtain the quantized weights can be performed using the following steps: S41, Input rescaling.

[0041] Before calculating the Hessian matrix, calibrate the input data. Multiply by the square root of the time importance fraction , obtain weighted input .

[0042] S42, matrix accumulation.

[0043] Constructing the Hessian matrix based on weighted input: The above embodiments inject temporal importance into the curvature information of the Hessian matrix, making the optimization direction more biased towards the time steps with higher weights.

[0044] S43. The time-weighted Hessian matrix is ​​used, and the optimal quantization weights are solved by Cholesky decomposition and inverse matrix solving method for the weighted least squares problem. and the corresponding quantization parameters.

[0045] The above embodiments, by employing a time-weighted Hessian matrix construction technique, can achieve priority fitting of static weights to dynamically generated trajectories, solving the problem that static parameters cannot simultaneously adapt to both high-noise and low-noise inputs, leading to the collapse of the generated structure at low bit depths. This ensures that even at aggressive compression ratios such as W3 / W4, the model can still maintain stable graphing and semantic alignment capabilities.

[0046] The above embodiments, by employing Cholesky decomposition and inverse matrix solving, can achieve low-cost and high-efficiency trajectory-aware weight calibration, solving the problem of excessively high computational costs caused by the introduction of temporal dependencies, which usually requires expensive nonlinear optimization or retraining.

[0047] To further reduce computational costs, in some specific embodiments of this application, S43, the quantization weights are solved by minimizing the weighted reconstruction error using the idea of ​​a second-order optimization algorithm. This specifically includes the following sub-steps: S431. Numerical stabilization of the Hessian matrix and Cholesky decomposition.

[0048] To ensure the stability of the numerical calculation, a small damping coefficient is first added to the diagonal elements of the time-weighted Hessian matrix obtained in step S42 to make it a positive definite matrix. Then, the modified matrix is ​​subjected to Cholesky decomposition: in, It is the identity matrix. It is a lower triangular matrix.

[0049] S432. Calculate the inverse of the Hessian matrix.

[0050] Using the results of Cholesky decomposition, the inverse of the Hessian matrix can be calculated efficiently. .because It is a lower triangular matrix, which can be solved using the forward substitution method, and thus obtained: S433. Column-by-column weight quantization and error compensation.

[0051] Column-wise traversal of the full-precision weight matrix The weights of each column are quantized, and the information of the inverse Hessian matrix obtained by S432 is used to update the remaining unquantized weights to compensate for the error caused by the current quantization.

[0052] Specifically, let's say we are currently processing the weights of the j-th column. The specific iterative process is as follows: S433.1, extract the j-th diagonal element from the inverse Hessian matrix. and the j-th column vector .

[0053] S433.2 quantizes the weight of the current column (e.g., rounds to the nearest quantized grid point) to obtain the quantized weight. Calculate the quantization error vector for this column: S433.3, based on the optimality condition of the second-order Taylor expansion, calculate the amount of update required for the remaining unquantized weights (i.e., from the (j+1)th column to the last column) in order to minimize the overall weighted error.

[0054] The formula for calculating the update vector is: S433.4, the calculated update vector is applied to the remaining weights, that is, all columns after column j are updated, thereby "pushing" the quantization error of the current column to the later columns for digestion: S433.5, after completing the compensation for the current column, update using Cholesky rank-1 update or by directly removing the current row / column. Prepare for the next iteration using the remaining submatrices.

[0055] Through iterative steps S433.1-S433.5, until all columns have been processed, a time-weighted quantization weight matrix is ​​finally obtained. This weight matrix is ​​applied at key time steps (corresponding to...). The larger the time interval, the smaller the activation value reconstruction error.

[0056] The above embodiments, by employing a combination of input rescaling and a standard second-order solver (GPTQ), can achieve low-cost and high-efficiency trajectory-aware weight calibration, solving the problem of excessively high computational costs caused by the introduction of temporal dependencies, which usually requires expensive nonlinear optimization or retraining.

[0057] Based on the same technical concept, another embodiment provides a virtual content generation method, including: Obtain virtual content generation instructions, the content of which describes information related to the virtual content to be generated; The diffusion model deployed on the smart device is invoked to generate corresponding virtual content based on the instructions. The diffusion model is calibrated by any of the time-sensitive diffusion model weight calibration methods.

[0058] In the above embodiments, since the quantization weights of the diffusion model have been optimized differently according to the importance of each denoising time step, it can still accurately capture the key structural evolution stages in the generation process under low bit deployment conditions, thereby significantly improving the semantic consistency, spatial layout rationality and detail fidelity of the generated virtual content, and effectively avoiding problems such as subject misalignment, structural collapse or content distortion caused by quantization distortion.

[0059] In some specific implementations, virtual content generation is the generation of paintings on mobile terminals, including: Receive drawing instructions from the user, which may include text descriptions or sketch information; The diffusion model deployed on the mobile terminal is invoked to generate image data based on drawing instructions, wherein the diffusion model is calibrated by any of the time-sensitive diffusion model weight calibration methods.

[0060] In some specific implementations, virtual content generation refers to the generation of virtual scenes within the in-vehicle system, including: Receive scene generation instructions, which include natural language voice instructions, driver's gaze or gesture recognition results, current navigation path information or vehicle surrounding environment perception data; The diffusion model deployed on the vehicle terminal is invoked to generate virtual scene data in real time based on scene generation instructions, which is used for augmented reality navigation, cockpit entertainment or human-computer interaction interface rendering. Among them, the diffusion model weight calibration method based on time-sensitive diffusion model is used for any term of the diffusion model.

[0061] Of course, the time-sensitive diffusion model weight calibration method can also be applied to various edge deployment scenarios involving dynamic generation processes and high computational cost models.

[0062] For example, when applied to textual video models (such as Stable Video Diffusion), temporal weighted calibration is used to maintain inter-frame consistency and temporal coherence; It is applied to speech synthesis systems (such as WaveGrad) to improve the preservation of audio details under low bit quantization. It is applied to medical image generation tasks (such as low-dose CT reconstruction and MRI super-resolution) to ensure the accurate reconstruction of key anatomical structures in a lightweight decoder. It can also be integrated into AR / VR or in-vehicle systems, supporting the real-time generation of high-quality virtual scenes on resource-constrained devices, meeting the needs for low-latency and highly immersive interaction.

[0063] The above scenarios all have high requirements for the structural rationality, temporal stability and inference efficiency of the generated results. The method in this application has the advantages of not requiring retraining, low sample dependence and strong generalization ability, and has wide applicability and potential for implementation.

[0064] The preferred features in the above embodiments can be used individually in any embodiment, or in any combination thereof, provided they do not conflict with each other. Furthermore, parts not described in detail in the embodiments can be implemented using existing technologies.

[0065] The following examples and comparative examples will be used to further illustrate this application in order to better understand the above-mentioned technical solutions. It should be understood that the following are only some examples and are not intended to limit this application.

[0066] Application Example 1: Achieving high-fidelity, low-bit virtual scene generation in an in-vehicle augmented reality navigation system.

[0067] This application example addresses the challenges of in-vehicle augmented reality (AR) navigation scenarios. It resolves the distortion issues that arise when in-vehicle terminals, due to limited computing power and storage resources, necessitate the use of a 3-bit (W3) aggressive quantization deployment diffusion model. These issues can lead to discrepancies between the displayed text and images, positional shifts, or structural breaks in virtual navigation elements such as lane lines, turn arrows, and construction warning signs. Ultimately, these distortions affect the reliability of the AR-HUD display and driving safety.

[0068] To ensure generation quality without increasing training costs and hardware burden, the system performs a time-sensitive weight calibration process before model deployment: S100 obtains the temporal importance weight. S101. Select 30 typical road scene images (such as intersections, construction zones, curves, etc.), input them into the pre-trained Flux.1-dev model, propagate forward to the output head layer, and perform a single backpropagation to calculate each denoising time step. t The corresponding Fisher score is used as a time-series sensitivity indicator. S102, the analysis results show that the key composition stage (such as...) t The Fisher Score of 50 is the initial stage (=50). t The fact that the time step is 5 times that of 10 indicates that this time step has a significant impact on the final output distribution. S103, normalize the Fisher Score for all time steps to obtain a normalized temporal importance score sequence { },satisfy =T, which gives higher weight to the time step during the plotting phase. >1), to achieve enhanced fitting of key stages.

[0069] S200, construct the time-weighted Hessian matrix.

[0070] S201, Read the input activation data corresponding to each time step. .

[0071] S202, inserts a preprocessing step into the traditional GPTQ calibration procedure: This step converts the original activation... Weighted rescaling: .

[0072] S203, using weighted activation to perform covariance accumulation: .

[0073] A weighted Hessian matrix reflecting temporal preferences is constructed, allowing the optimization direction to focus primarily on the high-sensitivity stage.

[0074] S300, Solving and Quantization Deployment S301, based on The optimal 3-bit quantization weights are solved by employing Cholesky decomposition and a layer-by-layer error minimization strategy. S302 saves the quantized weight parameters as a W3 format model file and deploys it to the vehicle terminal.

[0075] After deployment, when a vehicle detects a construction area or complex intersection ahead through the environmental perception module, the system calls the calibrated W3 model to generate semantically accurate and spatially aligned virtual guide lines and warning signs in real time, which are then overlaid onto the AR-HUD or central control screen, significantly improving the structural consistency, semantic accuracy, and human-machine interaction reliability of AR navigation on low-resource vehicle platforms.

[0076] This application example uses a Fisher information-based temporal importance assessment and normalization technique, which can accurately locate and weight key stages (such as the mapping period) in the generated trajectory, solving the problem that the fitting requirements of key time steps in the traditional mean calibration method are diluted by non-key time steps. Because of the use of time-weighted Hessian matrix construction technology, it is possible to prioritize the fitting of static weights to the dynamically generated trajectory, thus solving the problem that static parameters cannot simultaneously adapt to high-noise input and low-noise input, which leads to the collapse of the generated structure under low bit conditions. By employing a technique that combines input rescaling with a standard second-order solver (GPTQ), low-cost and high-efficiency trajectory-aware weight calibration can be achieved, solving the problem of excessively high computational costs caused by the expensive nonlinear optimization or retraining that is usually required when introducing temporal dependencies.

[0077] In summary, by employing the above-mentioned combined techniques, robust calibration of the static weights of the diffusion model can be achieved across the entire time series, overcoming the problems of the failure of the independent and identically distributed assumption, loss of graphing capability due to low bit quantization, and high computational overhead in the calibration process that exist in existing technologies.

[0078] Based on the same technical concept, in some specific embodiments of this application, a terminal includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and a method that the processor can use to execute when executing the program.

[0079] Optionally, the memory is used to store programs; the memory may include volatile memory, such as random-access memory (RAM), such as static random-access memory (SRAM), double data rate synchronous dynamic random-access memory (DDR SDRAM), etc.; the memory may also include non-volatile memory, such as flash memory. The memory is used to store computer programs (such as application programs and functional modules that implement the above methods), computer instructions, etc., and the aforementioned computer programs and computer instructions can be partitioned and stored in one or more memories. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by the processor.

[0080] The aforementioned computer programs, computer instructions, etc., can be stored in partitions within one or more memory locations. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by a processor.

[0081] A processor is used to execute a computer program stored in memory to implement the various steps of the methods involved in the above embodiments. For details, please refer to the relevant descriptions in the preceding method embodiments.

[0082] The processor and memory can be separate structures or integrated structures. When the processor and memory are separate structures, they can be coupled together via a bus.

[0083] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0084] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0085] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0086] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0087] The foregoing has described some specific embodiments of this application. It should be understood that this application is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the substantive content of this application. The above-described preferred features can be used in any combination without conflict.

Claims

1. A weight calibration method for a time-sensitive diffusion model, characterized in that, include: Using several calibration data, the sensitivity indices of each layer of the diffusion model at different time steps were calculated; The sensitivity of the entire time step is normalized to obtain a normalized temporal importance score; Based on the normalized temporal importance score, the weight quantization problem is modeled as a weighted least squares problem; The weighted least squares problem is solved to obtain the calibrated quantization weights.

2. The method for weight calibration of a time-sensitive diffusion model according to claim 1, characterized in that, The sensitivity metrics include Fisher information or gradient norm.

3. The method for weight calibration of a time-sensitive diffusion model according to claim 1, characterized in that, The weighted least squares problem is specifically as follows: ； in The original full-precision weights for the diffusion model. To quantify weights, For time step Input features, This represents the normalized temporal importance score.

4. The method for weight calibration of a time-sensitive diffusion model according to claim 1, characterized in that, Solving the weighted least squares problem to obtain quantized weights includes: Construct a time-weighted Hessian matrix; Using the Cholesky decomposition and inverse matrix solution method, based on the time-series weighted Hessian matrix, the optimal quantization weights and corresponding quantization parameters are solved for the weighted least squares problem.

5. The time-sensitive diffusion model weight calibration method according to claim 4, characterized in that, The construction of the time-weighted Hessian matrix includes: Calibration input data Multiply by the square root of the time importance fraction , obtain weighted input ; Construct the Hessian matrix based on the weighted input: ； Where T represents the total number of time steps in the diffusion model.

6. The method for weight calibration of a time-sensitive diffusion model according to claim 4, characterized in that, The step of solving the time-series weighted Hessian matrix using Cholesky decomposition and inverse matrix methods to find the optimal quantization weights and corresponding quantization parameters for the weighted least squares problem includes: Add damping terms to the diagonal elements of the time-weighted Hessian matrix to ensure its positive definiteness; The positive definite time-weighted Hessian matrix is ​​subjected to Cholesky decomposition to obtain the lower triangular matrix; Using the lower triangular matrix, the weighted Hessian inverse matrix is ​​solved by substitution. The weighted Hessian inverse matrix and the full-precision weight matrix are processed in column order: After quantization of the j-th column of the full-precision weight matrix, the compensation update amount to be applied to each subsequent column is calculated based on the information of the corresponding column in the weighted Hessian inverse matrix, and the compensation update amount is applied to offset the quantization error of the current column; then the remaining submatrix is ​​updated, and the next column iteration is started. The column-by-column iterative process is executed repeatedly until all columns have been processed. The resulting quantization weight matrix has the minimum weighted reconstruction error at the key time step and outputs the quantization parameters adapted to the weights simultaneously.

7. A method for generating virtual content, characterized in that, include: Obtain virtual content generation instructions, the content of which describes information related to the virtual content to be generated; The diffusion model deployed on the smart device is invoked, and corresponding virtual content is generated based on the instructions, wherein the diffusion model is calibrated by the time-sensitive diffusion model weight calibration method according to any one of claims 1-6.

8. The method for generating high-quality virtual content according to claim 7, characterized in that, The virtual content generation is a painting generation method for mobile terminals, including: Receive drawing instructions input by the user, the drawing instructions including text descriptions or sketch information; The diffusion model deployed on the mobile terminal is invoked to generate image data based on the drawing instructions, wherein the diffusion model is calibrated by the time-sensitive diffusion model weight calibration method according to any one of claims 1-6.

9. The method for generating high-quality virtual content according to claim 7, characterized in that, The virtual content generation is a virtual scene generation for the in-vehicle system, including: Receive scene generation instructions, which include natural language voice instructions, driver's gaze or gesture recognition results, current navigation path information or vehicle surrounding environment perception data; The diffusion model deployed on the vehicle terminal is invoked to generate virtual scene data in real time based on the scene generation instructions, which is used for augmented reality navigation, cockpit entertainment or human-computer interaction interface rendering. The diffusion model is calibrated using the time-sensitive diffusion model weight calibration method described in any one of claims 1-6.

10. A terminal, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, can be used to perform the method of any one of claims 1-6.

Images