Reasoning methods and electronic devices

Abstract

This invention discloses an inference method and electronic device, relating to the field of model inference technology. The method is applied to a pre-set accelerator and includes calculating the rate of change of data features, the coefficient of variation of the rate of change of data features, and the cumulative deviation within the target sliding window based on input data and inference output data. An initial decision threshold is calculated based on the rate of change of data features, the coefficient of variation, and the cumulative deviation. A final decision threshold is determined based on the initial decision threshold and the current graphics processor utilization. A cache reuse operation or a full inference operation is performed based on the final decision threshold. Thus, by dynamically adjusting the decision thresholds for cache reuse and full inference, manual threshold calibration is avoided, allowing the thresholds to adapt to changes in computing resources. This reduces the computational load on the graphics processor during cache reuse and ensures output accuracy during full inference.

Description

Technical Field

[0001] This invention relates to the field of model reasoning technology, and more particularly to reasoning methods and electronic devices. Background Technology

[0002] In existing technologies, the thresholds for reused data inference and full inference in video generation models need to be calibrated individually for each model or dataset, which increases the cost of use. Furthermore, if a fixed threshold is set, it cannot respond to real-time changes in computing resource utilization. Summary of the Invention

[0003] This invention provides an inference method and electronic device to at least solve the problems in related technologies where the threshold of video diffusion models needs to be individually calibrated for each model, increasing the cost of use, and where fixed thresholds cannot respond to real-time changes in computing resource utilization.

[0004] This invention provides an inference method applied to a preset accelerator. The method includes: acquiring input data and inference output data within a target sliding window; calculating, based on the input data and the inference output data, the rate of change of data features within the target sliding window, the coefficient of variation of the rate of change of data features, and the cumulative deviation; calculating an initial decision threshold based on the rate of change of data features, the coefficient of variation, and the cumulative deviation of the data feature changes; determining a final decision threshold based on the initial decision threshold and the current graphics processor utilization; and performing a cache reuse action or a full inference action based on the final decision threshold.

[0005] The present invention also provides an inference system for a preset accelerator. The system includes: an acquisition module for acquiring input data and inference output data within a target sliding window; a calculation module for calculating, based on the input data and the inference output data, the rate of change of data features within the target sliding window, the coefficient of variation of the rate of change of data features, and the cumulative deviation; and an inference module for calculating an initial decision threshold based on the rate of change of data features, the coefficient of variation, and the cumulative deviation of the data feature changes, determining a final decision threshold based on the initial decision threshold and the current graphics processor utilization, and performing a cache reuse action or a full inference action based on the final decision threshold.

[0006] The present invention also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-described reasoning methods when executing the computer program.

[0007] The present invention also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-described reasoning methods.

[0008] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described reasoning methods.

[0009] This invention calculates the rate of change of data features, the coefficient of variation of the rate of change of data features, and the cumulative deviation within the target sliding window. An initial decision threshold is calculated based on these metrics. The final decision threshold is then determined based on the initial decision threshold and the current graphics processor utilization. A cache reuse or full inference operation is then performed based on the final decision threshold. This solves the technical problems of video diffusion models requiring individual threshold calibration for each model, increasing usage costs, and the inability of fixed thresholds to respond to real-time changes in computing resource utilization. The invention achieves the technical effect of dynamically adjusting the decision thresholds for cache reuse and full inference, avoiding manual calibration, and making the thresholds adaptable to changes in computing resources. It reduces the computational load on the graphics processor during cache reuse and ensures output accuracy during full inference. Attached Figure Description

[0010] To more clearly illustrate the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 A flowchart of a reasoning method provided in an embodiment of the present invention; Figure 2 A flowchart of cache reuse inference is provided for an embodiment of the present invention; Figure 3 A flowchart of full inference provided for an embodiment of the present invention; Figure 4 This is a schematic diagram of the inference system provided in an embodiment of the present invention. Detailed Implementation

[0012] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the present invention.

[0013] It should be noted that, in the description of this invention, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., used in this invention are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0014] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0015] Video generation technology is a core research direction in the field of generative models, widely used in digital content creation, virtual environment interaction, and many other scenarios. Compared to earlier video generation methods based on the traditional U-Net architecture, Diffusion Transformer (DiTs) models have become the mainstream paradigm for video generation due to their superior spatiotemporal feature representation capabilities and scalability, and are widely used in mainstream video generation models such as Sora, HunyuanVideo, and Wan2.1. These models model the complex spatiotemporal dependencies of videos through iterative denoising mechanisms, enabling the generation of high-quality video content. However, the iterative denoising mechanism has fatal flaws: slow inference speed and extremely high computational resource consumption. For example, generating a 5-second 720P video using an H2O GPU takes approximately 2 hours, severely limiting its practical application in real-time scenarios.

[0016] To overcome this technical barrier, various optimization schemes are proposed, including model distillation, architecture optimization, and training-free acceleration. A representative existing technology is the EasyCache adaptive caching framework. This scheme analyzes the feature change patterns during the DiT model denoising process, leverages the characteristic that the transformation rate tends to stabilize in the later stages of denoising, sets a dynamic threshold judgment mechanism, adaptively reuses historical computational features, reduces redundant computation, and accelerates model inference.

[0017] However, this existing technology still has significant drawbacks. First, its judgment threshold needs to be calibrated separately for different models and datasets, resulting in high adaptation costs and poor universality. Second, fixed calibration thresholds cannot adapt to the real-time fluctuations in device computing resources and are difficult to dynamically adapt to the operating environment. Third, independent monitoring and cache reuse modules will additionally occupy inference computing power, further compressing acceleration gains and failing to achieve efficient and universal video diffusion model accelerated inference.

[0018] The embodiments of the present invention provide a reasoning method, and the method is described in detail in conjunction with the execution flow of the reasoning method.

[0019] The inference method is applied to a preset accelerator, which can be a field-programmable gate array (FPGA), a dedicated AI accelerator, a microcontroller (MCU), or an embedded coprocessor, without any specific limitations.

[0020] In step S101, the input data and inference output data within the target sliding window are acquired.

[0021] In step S102, based on the input data and the inference output data, the rate of change of data features, the coefficient of variation of the rate of change of data features, and the cumulative deviation within the target sliding window are calculated.

[0022] In some embodiments, the target sliding window includes at least one time step. Based on input data and inference output data, calculating the rate of change of data features within the target sliding window includes: calculating the input data deviation and inference output data deviation at least some time steps within the target sliding window based on the input data and inference output data; and obtaining the rate of change of data features within the target sliding window at least some time steps based on the ratio of the inference output data deviation at least some time steps to the input data deviation at the corresponding time step.

[0023] Among them, the rate of change of data features is from The calculation yields the following, where the input data deviation at each time step is: The inference output data bias at each time step is .

[0024] Understandably, for the target time sliding window At each time step t, the deviation of the input data and the deviation of the inference output data are calculated respectively. Based on the ratio of the deviation of the inference output data to the deviation of the input data, the rate of change of the data features at step t is obtained.

[0025] For example, a sliding window with a target time. For example, there are 3 time steps: t1, t2, and t3.

[0026] The deviation of the input data for t1 is 2, and the deviation of the inference output data is 4. The deviation of the input data for t2 is 3, and the deviation of the inference output data is 1.5. The deviation of the input data for t3 is 1, and the deviation of the inference output data is 5. Therefore, the rate of change of the data characteristic of t1 is: 4 / 2 = 2; The rate of change of the data characteristics of t2: 1.5 / 3 = 0.5; The rate of change of the data characteristics of t3 is 5 / 1 = 5.

[0027] Thus, within the target time sliding window, the rate of change of data characteristics at three time points is obtained.

[0028] The above technical solution quantifies the fluctuation of data characteristics at a single time step by using the ratio of the deviation of the inference output data to the deviation of the input data, providing a reliable quantitative basis for determining the cache reuse threshold.

[0029] In some embodiments, the coefficient of variation of the rate of change of data features within the target sliding window is calculated based on input data and inference output data, including: calculating the mean and standard deviation of the rate of change of data features within the target sliding window based on the rate of change of data features at least some time steps; and obtaining the coefficient of variation of the rate of change of data features within the target sliding window based on the ratio of the mean of the rate of change of data features at least some time steps to the standard deviation of the rate of change of data features at the corresponding time steps.

[0030] The coefficient of variation of the rate of change of data characteristics is from Calculated.

[0031] Specifically, after calculating the rate of change of data features at each time step, the mean of the rate of change of data features within the target sliding window is calculated. and the standard deviation of the rate of change of data features within the target sliding window ,according to and The coefficient of variation of the rate of change of data features within the target sliding window is obtained by calculating the ratio.

[0032] For example, the target time sliding window described above Taking three time steps as an example: t1, t2, and t3, the data feature change rate of t1 is 2; the data feature change rate of t2 is 0.5; and the data feature change rate of t3 is 5. Then, the mean of the data feature change rate within the target sliding window is... The standard deviation of the rate of change of data features within the target sliding window is 5. If the value is approximately 2.3, then the coefficient of variation of the rate of change of data features within the target sliding window is... The value is 2.3 / 5 = 0.46.

[0033] The above technical solution uses the ratio of the mean to the standard deviation of the rate of change of data features at each time step to obtain the coefficient of variation of the rate of change of data features, which characterizes the relative fluctuation of the rate of change of data features within the time window, providing a precise basis for the dynamic adjustment of the adaptive threshold.

[0034] In some embodiments, calculating the cumulative deviation of data feature changes within a target sliding window based on input data and inference output data includes: calculating the deviation of input data at least for a portion of time steps within the target sliding window; calculating the product of the deviation of input data at least for a portion of time steps within the target sliding window and the rate of change of data features at least for a portion of time steps within the target sliding window; calculating the ratio of the product to the corresponding inference output data of the input data at least for a portion of time steps to obtain the ratio corresponding to at least a portion of time steps; and summing the ratios corresponding to at least a portion of time steps to obtain the cumulative deviation of data feature changes within the target sliding window.

[0035] The deviation of the input data at each time step is: The input data at each time step corresponds to the inference output data as follows: .

[0036] The cumulative deviation of data feature changes is from Calculated.

[0037] The deviation of the input data calculated at each time step is: Rate of change of data features at time steps The product of these products is then compared with the inference output data to obtain the ratio at each time step. This allows for the summation of the ratios at each time step, yielding the cumulative deviation of data feature changes within the target sliding window.

[0038] For example, K 1 = 2, K 2 = 0.5, K 3 = 5. The input data deviation for t1 is 2, the input data deviation for t2 is 3, and the input data deviation for t3 is 1. The inference output data... V 1 is 4, V 2 is 3, V 3 is 1.

[0039] Then the ratio of t1, t2, t3 is determined by The calculation shows that the ratios of t1, t2, and t3 are 1, and the cumulative deviation of the data characteristic changes is 3.

[0040] By integrating the deviation of input data, the rate of change of features, and output data, the overall deviation of data in the iterative process is accumulated, providing a basis for starting and stopping cache reuse.

[0041] In step S103, an initial decision threshold is calculated based on the data feature change rate, coefficient of variation, and cumulative deviation. A final decision threshold is determined based on the initial decision threshold and the current graphics processor utilization. A cache reuse action or a full inference action is then performed based on the final decision threshold.

[0042] In some embodiments, calculating an initial decision threshold based on the rate of change of data features, the coefficient of variation, and the cumulative deviation includes: calculating a first product of the coefficient of variation of the rate of change of data features and a first scaling factor; determining the instantaneous derivative of the cumulative deviation of data feature changes and calculating a second product of the instantaneous derivative of the cumulative deviation and a second scaling factor; and calculating the initial decision threshold based on the first product and the second product.

[0043] Specifically, the initial decision threshold is determined by The calculation yielded the following result. It is an initial threshold (e.g., 5%), used as a baseline. Within the current sliding window Coefficient of variation (window size can be set) =10 steps). yes The instantaneous derivative is calculated through adjacent steps within the sliding window. It is the first scaling factor. It is the second scaling factor (experimental calibration required, for example) =0.3, =0.1), controlling sensitivity.

[0044] By integrating the two-dimensional parameters of data feature change rate and cumulative deviation of data feature change, and combining them with the benchmark threshold calculation, the initial judgment threshold can be dynamically generated, eliminating the need for manual calibration.

[0045] In some embodiments, determining the final decision threshold based on the initial decision threshold and the current graphics processor utilization includes: determining the utilization range in which the current graphics processor utilization is located; determining the threshold adjustment coefficient corresponding to the initial decision threshold based on the utilization range in which the current graphics processor utilization is located; and obtaining the final decision threshold by multiplying the threshold adjustment coefficient and the initial decision threshold.

[0046] The final judgment threshold is: , This is the threshold adjustment coefficient.

[0047] In this step, the current GPU utilization is first obtained in real time to determine its utilization range.

[0048] The initial decision threshold is determined based on the utilization rate range. Determine the corresponding threshold adjustment coefficient .

[0049] The obtained threshold adjustment coefficient With the initial decision threshold Multiply them to obtain the final decision threshold under the current graphics processor utilization.

[0050] By using the above technical solution, the initial decision threshold is dynamically corrected in combination with the load status of the image processor, so that the final decision threshold is adapted to the real-time computing power conditions, avoiding idle computing power waste or overload lag.

[0051] In some embodiments, determining the threshold adjustment coefficient corresponding to the initial decision threshold based on the current utilization range of the graphics processor includes: in response to the current graphics processor utilization being in a first utilization range, using the first coefficient as the threshold adjustment coefficient corresponding to the initial decision threshold; in response to the current graphics processor utilization being in a second utilization range, calculating the second coefficient based on a preset linear relationship according to the current graphics processor utilization, and using the second coefficient as the threshold adjustment coefficient corresponding to the initial decision threshold, wherein the lower limit of the second utilization range is greater than the upper limit of the first utilization range, and the second coefficient is greater than the first coefficient; in response to the current graphics processor utilization being in a third utilization range, using the third coefficient as the threshold adjustment coefficient corresponding to the initial decision threshold, wherein the lower limit of the third utilization range is greater than the upper limit of the second utilization range, and the third coefficient is greater than the second coefficient.

[0052] In some embodiments, the preset linear relationship is: ; in, This is the threshold adjustment coefficient. This represents the current graphics processor utilization rate.

[0053] For example, taking the first utilization interval as... The second utilization rate range is The third utilization rate range is Let's take an example to illustrate.

[0054] The threshold adjustment coefficient corresponding to the initial decision threshold is determined by: Sure.

[0055] In other words, when the current GPU utilization is detected to be less than 50%, the following operation is performed: The first coefficient (i.e., 1 in the function) is used as the threshold adjustment coefficient corresponding to the initial decision threshold. At this time, computing resources are sufficient, and there is no need to sacrifice image quality for speed. By "tightening" the threshold, the judgment standard for cache reuse is improved, unnecessary cache usage is reduced, and priority is given to ensuring the accuracy of model inference and the image quality of the generated video.

[0056] When it is detected that 50% ≤ current GPU utilization < 80%, perform the following operation: Based on a preset linear relationship, a second coefficient is determined according to the current graphics processor utilization. The calculated second coefficient is greater than 1 and is used as the threshold adjustment coefficient corresponding to the initial decision threshold. That is, within this range, for every 1% increase in utilization, the coefficient increases by 0.046, thus gradually widening the threshold.

[0057] By using a linearly increasing adjustment coefficient, the threshold is gradually widened as the computing load increases, thus balancing image quality and inference speed while ensuring a certain level of image quality and gradually increasing the proportion of cache reuse.

[0058] When the current GPU utilization is detected to be ≥80%, the following operations are performed: The third coefficient (i.e., 2.25 in the function) is used as the threshold adjustment coefficient corresponding to the initial decision threshold. This coefficient is 2.25, which is much larger than the coefficient value of the second utilization interval, thus achieving a significant relaxation of the threshold.

[0059] At this time, computing resources are scarce. In order to ensure that inference can be completed within an acceptable time, the judgment criteria for cache reuse are greatly relaxed, the cache reuse ratio is significantly increased, and a higher inference speedup ratio is obtained by sacrificing a small loss in image quality, so as to ensure the real-time response capability of the system.

[0060] Through the above technical solutions, when GPU utilization is in the low range, a smaller coefficient is used to convergently adjust the initial decision threshold, reducing cache reuse frequency, prioritizing the accuracy of iterative inference results, and avoiding unnecessary precision loss. When GPU utilization is in the middle range, an adjustment coefficient is calculated based on a linear correlation formula. The coefficient changes smoothly and synchronously with the utilization rate, allowing for gradual adjustment of the threshold according to the computing power occupancy, resulting in a natural and abrupt transition between states. When GPU utilization is in the high range, a larger fixed coefficient is used to significantly relax the decision threshold, increase the probability of reusing intermediate calculation results, effectively reduce the computational load, alleviate GPU computational pressure, and prevent inference stuttering and task timeouts.

[0061] In some embodiments, performing a cache reuse action or a full inference action based on a final decision threshold includes: performing a cache reuse action or a full inference action based on a final decision threshold and a cumulative deviation.

[0062] In some embodiments, performing a cache reuse action or a full inference action based on a final decision threshold and a cumulative deviation includes: performing a cache reuse action in response to a cumulative deviation being less than a final decision threshold; and performing a full inference action in response to a cumulative deviation being greater than or equal to a final decision threshold.

[0063] Specifically, when cumulative deviation is detected Less than the final judgment threshold e When the data changes within the current model are minimal, the iterative computation is highly redundant, and there is no need to perform a full forward inference computation. In this case, the full computation of the GPU is skipped, and the iterative computation is completed using hardware accelerators. The computation results from the previous time step are reused as the inference output of the current time step, significantly reducing GPU computing power consumption and inference latency.

[0064] When cumulative deviation is detected E t Greater than or equal to the final decision threshold e If the system determines that the data feature transformation rate of the model fluctuates drastically during the current inference stage, and the historical cached data can no longer accurately fit the current model output, the system will immediately terminate the cache reuse logic to avoid video generation quality distortion and feature loss. Instead, it will use the output data of the current time step as the input for the next inference step, driving the GPU to perform full inference. Full inference ensures the accuracy of video generation.

[0065] Through the above technical solution, when the cumulative deviation is less than the threshold, cache reuse is enabled, skipping the heavy full forward computation of the GPU. Relying on the hardware accelerator to lightweight inference results, the computational load of model iteration is greatly reduced, and the inference latency and GPU computing power consumption are significantly reduced, solving the pain points of slow inference speed and high computing power consumption of traditional video diffusion models. When the cumulative deviation exceeds the standard, cache reuse is immediately stopped and full inference is forced to be enabled. The spatiotemporal feature calculation process of the model is fully preserved, effectively avoiding the accuracy degradation problems such as image distortion, inter-frame jitter, and feature loss caused by cache reuse. It can speed up stable scenes and maintain accuracy in sudden scenes. While realizing intelligent adaptive inference, it ensures the efficient and smooth operation of the video generation inference process.

[0066] In some embodiments, performing a cache reuse operation includes: calculating the difference between the inference output data and the input data corresponding to the current time step; determining the input data for the next time step based on the inference output data corresponding to the current time step; and obtaining the inference output data for the next time step based on the input data and the difference for the next time step.

[0067] The system relies on hardware accelerators to complete lightweight inference without GPU full computation. Specific cache reuse steps are as follows: Figure 2 As shown: First, calculate the difference between the inference output data and the input data corresponding to the current time step.

[0068] The hardware accelerator reads the output data generated by the GPU's full inference at the current time step t. x i and the inference output data for the corresponding time step. v i The difference Δ between the inference output data and the input data at the current time step is obtained through difference operations.i = v i -x i .

[0069] Furthermore, the input data for the next time step is determined. The system will then output the full inference data for the current time step. v i The input data for the next inference time step t+1 is directly assigned. x i+1 This completes the update of the iterative input.

[0070] Finally, the inference output data for the next time step is deduced. The difference Δ between the inference output data and the input data at the next inference time step and the current time step is calculated. i = v i -x i The inference output data for the next time step is derived. v i+1 The specific calculation logic is as follows: v i+1 = x i+1 +Δ i .

[0071] After completing a single step of cache reuse inference, the system will automatically update the feature change rate within the sliding window. Cumulative deviation Coefficient of variation Monitored variables, and then compare them with dynamic decision thresholds. The comparison is performed to determine whether to perform cache reuse or full inference.

[0072] The above technical solution relies on the difference between the inference output data and the input data corresponding to the current time step, directly assigns the inference output data of the current time step as the input data of the next inference time step, and performs a summation operation to complete the inference. This eliminates the need to call the GPU to perform complex calculations, significantly reducing the amount of computation and effectively reducing hardware computing power overhead.

[0073] In some embodiments, before performing the cache reuse action, the method further includes: identifying whether the inference output data corresponding to the current time step meets the preset standard output conditions; if the inference output data corresponding to the current time step does not meet the preset standard output conditions, stopping the cache reuse action and performing a full inference action.

[0074] Specifically, before performing cache reuse, this invention adds a pre-compliance verification mechanism. This mechanism uses preset standard output conditions to perform validity screening, thereby achieving controllable and secure execution of cache reuse. The specific logic is as follows: Before each cache reuse action is triggered, the accelerator monitoring module identifies and verifies the inference output data corresponding to the current time step, determining whether the inference output data meets the preset standard output conditions. The preset standard output conditions are quantization constraints pre-defined based on the characteristics of the video generation model, such as the coefficient of variation. Within the normal fluctuation range, data characteristic change rate No mutations or other indicators were found.

[0075] If the verification result shows that the inference output data at the current time step meets the standard output conditions, it proves that the current model inference state is stable and the feature transformation law is stable. The subsequent cache reuse inference action will be executed normally to achieve lightweight and accelerated inference.

[0076] If the verification result indicates that the output data of the current time step inference does not meet the standard output conditions, it proves that there is instability such as feature mutation or abnormal noise fluctuation in the current inference stage. If cache reuse continues, it may introduce precision errors, leading to problems such as blurred video and frame jumps. At this time, the cache reuse action will be stopped immediately, and full inference will be switched to generate accurate output data through complete iterative denoising calculation by the GPU, ensuring the stability of video generation quality.

[0077] By verifying the inference output data of the previous time step before cache reuse, unstable conditions such as feature mutations and data anomalies can be identified in advance, unreasonable cache calls can be eliminated from the source, error accumulation can be reduced, and image quality defects such as blurry images and frame jumps can be effectively avoided.

[0078] In some embodiments, performing a full inference action includes: determining the model parameters and total number of time steps corresponding to the current inference process; determining the input data for the next time step based on the inference output data corresponding to the current time step; and outputting the inference output data for the next time step based on the input data for the next time step, the model parameters corresponding to the current inference process, and the total number of time steps.

[0079] When the cumulative deviation is greater than or equal to the final decision threshold, cache reuse will be exited and full inference will be switched to. The steps of full inference are as follows: Figure 3 As shown: Step 1: Input data x i Access the GPU or accelerator device and perform full inference of the diffusion model. First, determine the model parameters θ and the total number of time steps corresponding to the current inference process. T After initiating the full inference branch, the system first reads the total number of time steps T and model parameters θ, such as model weights, bias parameters, feature dimension parameters, and temporal modeling parameters.

[0080] The second step is to determine the input data for the next time step based on the inference output data of the current time step. This involves processing the inference output data obtained from the full model calculation at the current time step t. v i Hardware accelerator reading v i The input data for the next inference time step t+1 is directly assigned. x i+1 .

[0081] The third step is based on the model parameters θ and the total number of time steps. T Generate inference output data for the next time step. Invoke the GPU to perform forward inference operations, taking the input data for the next time step... x i+1 The model parameters θ and the total number of time steps in the current inference process. T Common input to diffusion model u Through the complete computational process of spatiotemporal feature extraction, noise prediction, and denoising correction, the model accurately outputs the inference output data at time step t+1. v i+1 The specific calculation logic is as follows: v i+1= u θ (x i+1 |T) This process does not skip any model calculation modules or use any historical cached differences. It relies entirely on the model's native calculation logic to complete the inference, effectively solving the video distortion problem during periods of drastic feature fluctuations.

[0082] After completing the full inference, the system will update the rate of change of data features within the sliding window. Cumulative deviation Coefficient of variation By monitoring and statistical variables, refreshing the judgment threshold, and entering the next round of dual-branch logical judgment, the adaptive alternation between full inference and cache reuse is achieved.

[0083] By employing the above technical solution, the parameters and iteration steps of the diffusion model are retrieved for inference, ensuring the complete continuity of data feature information in the preceding and following steps and guaranteeing the accuracy of the inference results.

[0084] In some embodiments, when stopping cache reuse if the inference output data corresponding to the current time step does not meet the preset standard output conditions, the method includes: generating a reminder message that the inference output data corresponding to the current time step does not meet the preset standard output conditions and stopping cache reuse.

[0085] If the inference output data at the current time step does not meet the preset standard output conditions, the system will immediately trigger the cache reuse termination response logic and proactively generate corresponding exception alert information. The alert information includes core data such as the current abnormal inference time step, real-time GPU resource utilization, coefficient of variation, data feature change rate, cumulative deviation value, and decision threshold, and records the triggering reason and system status of this cache reuse termination.

[0086] While generating the alert message, the cache reuse action is stopped, and the full inference mode is switched to. The GPU full model calculation is used to replace the lightweight cache inference, thus preventing image quality degradation and inference bias caused by the accumulation of abnormal features.

[0087] By using the above technical solution, when the inference output data corresponding to the current time step does not meet the preset standard output conditions, the time step, hardware load, various characteristic indicators and threshold parameters are summarized to facilitate quick troubleshooting of inference anomalies and generate reminders, making it easier for operations and maintenance to grasp the inference running status and improve process monitorability.

[0088] It should be noted that although the reasoning method of this invention is designed and optimized with the video generation model based on the DiT diffusion transformer as the core application scenario, its core technical logic is not limited to the field of video generation. It can be transferred and adapted to various iterative numerical simulation, real-time perception reasoning, distributed iterative computing and other technical fields. The following will elaborate on the typical extended scenarios, specific adaptation schemes and optimization logic.

[0089] 1. For iterative numerical simulation fields such as fluid and climate simulations Unlike video generation, which calculates decision thresholds based on the rate of change of data features, fluid and climate numerical simulation scenarios can construct threshold systems based on the dynamic changes of core physical quantities. Taking a typhoon motion simulation scenario as an example, the core monitored physical quantities are wind speed, wind direction, air pressure, and flow velocity. The specific calculation process is as follows: Wind speed was selected as the core monitoring indicator. In each computation node of the simulation grid, wind speed magnitude and direction data were collected through a sliding window for consecutive iteration steps, and the rate of change of wind speed in adjacent iteration steps was calculated. The rate of change of wind speed in the eye of the typhoon and the core area of ​​the storm was extremely large, and the physical state fluctuated violently; the rate of change of wind speed in the outer calm sea area and non-storm area was very small, and the state tended to be stable.

[0090] Based on the multi-step wind speed change rate data within a preset sliding window, the mean and standard deviation of the change rate within the window are statistically analyzed, and the coefficient of variation of the wind speed change rate is further calculated. The wind speed calculation deviation of each iteration step within the sliding window is also accumulated.

[0091] By combining the initial baseline threshold, wind speed variation coefficient, wind speed change rate, wind speed calculation deviation, and fixed scaling factor, an adaptive threshold calculation formula is reconstructed to fit the fluid simulation scenario.

[0092] (2) This invention can also be extended to distributed systems to realize dynamic caching strategy adjustment based on node load.

[0093] When the system detects that the node load is low and the computing resources are sufficient, it automatically tightens the cache reuse threshold and reduces the cache reuse frequency to prioritize the accuracy of numerical simulation. When the system detects that the node load is full and the computing resources are exhausted, making it unable to support high-frequency, high-precision full-scale iterative calculations, it actively relaxes the adaptive threshold, forcibly increases the cache reuse frequency, reuses historical intermediate calculation results on a large scale, reduces the computing pressure on the nodes, avoids node computing power overload, iteration lag, simulation interruption and other problems, and ensures that the distributed simulation task can converge quickly and output stable results.

[0094] (3) This invention can also be extended to in-vehicle autonomous driving systems.

[0095] Vehicle-mounted CPUs, GPUs, and NPUs have limited computing power resources, yet they need to simultaneously handle multiple tasks such as autonomous driving perception, path planning, vehicle control, voice interaction, and in-vehicle world model diffusion generation, leading to fierce competition for computing resources. This invention's lightweight monitoring and resource-aware adaptive caching mechanism is adaptable to in-vehicle edge inference scenarios, ensuring the real-time performance and stability of in-vehicle intelligent inference through dynamic cache scheduling.

[0096] In automotive scenarios, real-time monitoring of key load metrics such as the utilization rate of the onboard GPU, NPU, and CPU, inference single-step time, and task queue backlog length is performed. Based on the real-time computing load, the cache reuse threshold and frequency are dynamically adjusted to achieve low-latency real-time response. When the utilization rate of the vehicle's GPU / NPU is below 80%, computing resources are sufficient, and inference time is ample, the system maintains a baseline caching strategy. For example, by default, cache reuse is performed once every 3 iterations, and full inference is performed in the remaining steps to balance inference accuracy and speed, and to ensure the image quality and detection accuracy of vehicle perception and world model generation.

[0097] When the onboard GPU / NPU load exceeds 80%, or when the current iteration inference time is about to exceed the onboard system's real-time threshold, posing a risk of stuttering and timeout, the adaptive decision threshold is dynamically relaxed, significantly increasing the cache reuse frequency. The cache reuse frequency is dynamically adjusted from once every 3 steps to once every 2 steps. In extremely high-load scenarios, cache reuse is forced every single step, maximizing the skipping of redundant computations and reducing inference latency.

[0098] To enable those skilled in the art to further understand the reasoning method of the embodiments of the present invention, a detailed description is provided below with reference to specific embodiments.

[0099] Starting at time step t=i, the inference chain is divided into two paths: a full inference branch and a cache reuse branch. Relying on a lightweight monitoring module and a hardware accelerator for collaborative scheduling, low-latency adaptive inference is achieved, and the entire process is iterated in a closed loop until the task terminates.

[0100] Step 1: Initialize the original input data for full inference at time t=i. x i The data is sent to the GPU hardware accelerator to initiate a full inference operation of the diffusion model, and the inference result at the current time step is output. v i .

[0101] Hardware accelerator reads inference results v i Synchronous operation monitoring characteristic parameters: data characteristic change rate Cumulative deviation Coefficient of variation .

[0102] By combining the real-time GPU load status, a dynamic decision threshold is generated through an adaptive threshold algorithm. e This serves as the criterion for determining branch switching.

[0103] Step 2: Lightweight State Monitoring and Branch Decision Making The lightweight monitoring module collects GPU utilization hardware status data in real time and identifies the rate of change of data characteristics. Cumulative deviation Coefficient of variation As input, after completing the feature weighting operation, the deviation E will be accumulated in real time. t With dynamic decision threshold e Perform a size comparison to trigger branching and traffic splitting: Branch A: < e Perform cache reuse inference: Calculate the current step deviation increment: Δ i = v i -x i 。

[0104] The inference output for the next time step is generated iteratively using the deviation: v i+1 = x i+1 +Δ i。

[0105] Hardware accelerator based on new output v i+1The feature parameters are updated again, the cumulative deviation is compared with the decision threshold again, and the inference is reused cyclically; once the condition is met... ≥ e Immediately jump to the full reasoning branch.

[0106] Branch B: ≥ e Perform a full inference restart and use the current inference result as the raw input for the next time step: x i+1 = v i ,Will x i+1 The data is fed into the GPU, and the full inference computation of the diffusion model is re-executed: v i+1= u θ (x i+1 |T)。

[0107] Synchronize back to the hardware parameter calculation, status monitoring, and branch determination process.

[0108] Step 3: Iterative Loop and Task Termination If cache reuse is performed: continuous computation occurs within the accelerator. v i Each step updates the rate of change of data features. Cumulative deviation Coefficient of variation and judge and e The size, until ≥ e When that happens, switch to the full inference branch.

[0109] If performing full inference: x i+1 =v i Enter GPU and execute again v i+1= u θ (x i+1 |T)， Repeat the "monitoring, decision-making, branching" logic of step 2 until the reasoning task ends.

[0110] The entire process iterates repeatedly until all time-step inference tasks are completed, at which point the process terminates and the final result is output.

[0111] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0112] Embodiments of the present invention also provide a reasoning system 10, such as Figure 4 As shown, the inference system 10 is applied to a preset accelerator and includes: an acquisition module 100 for acquiring input data and inference output data within a target sliding window; a calculation module 200 for calculating the rate of change of data features, the coefficient of variation of the rate of change of data features, and the cumulative deviation within the target sliding window based on the input data and the inference output data; and an inference module 300 for calculating an initial decision threshold based on the rate of change of data features, the coefficient of variation, and the cumulative deviation, determining a final decision threshold based on the initial decision threshold and the current graphics processor utilization, and performing a cache reuse action or a full inference action based on the final decision threshold.

[0113] In some embodiments, the target sliding window includes at least one time step, and the calculation module 200 is configured to: calculate the input data deviation and inference output data deviation of at least a portion of the time steps within the target sliding window based on input data and inference output data; and obtain the data feature change rate of at least a portion of the time steps within the target sliding window according to the ratio of the inference output data deviation of at least a portion of the time steps to the input data deviation of the corresponding time step.

[0114] In some embodiments, the calculation module 200 is configured to: calculate the mean and standard deviation of the rate of change of data features within a target sliding window based on the rate of change of data features at least at some time steps; and obtain the coefficient of variation of the rate of change of data features within the target sliding window based on the ratio of the mean of the rate of change of data features at least at some time steps to the standard deviation of the rate of change of data features at the corresponding time steps.

[0115] In some embodiments, the calculation module 200 is configured to: calculate the deviation of the input data at least a portion of the time steps within the target sliding window; calculate the product of the deviation of the input data at least a portion of the time steps within the target sliding window and the rate of change of the data features at least a portion of the time steps within the target sliding window; calculate the ratio of the product to the inference output data corresponding to the input data at least a portion of the time steps to obtain the ratio corresponding to at least a portion of the time steps; and calculate the cumulative deviation of the data feature changes within the target sliding window by summing the ratios corresponding to at least a portion of the time steps.

[0116] In some embodiments, the inference module 300 is configured to: calculate the coefficient of variation of the rate of change of data features and the first product of the first scaling factor; determine the instantaneous derivative of the cumulative deviation of the data feature changes and calculate the second product of the instantaneous derivative of the cumulative deviation and the second scaling factor; and calculate an initial decision threshold based on the first product and the second product.

[0117] In some embodiments, the inference module 300 is configured to: determine the utilization range in which the current graphics processor utilization is located; determine the threshold adjustment coefficient corresponding to the initial decision threshold based on the utilization range in which the current graphics processor utilization is located; and obtain the final decision threshold by multiplying the threshold adjustment coefficient and the initial decision threshold.

[0118] In some embodiments, the inference module 300 is configured to: respond to the current graphics processor utilization being in a first utilization range, use a first coefficient as a threshold adjustment coefficient corresponding to an initial decision threshold; respond to the current graphics processor utilization being in a second utilization range, calculate a second coefficient based on a preset linear relationship according to the current graphics processor utilization, and use the second coefficient as a threshold adjustment coefficient corresponding to the initial decision threshold, wherein the lower limit of the second utilization range is greater than the upper limit of the first utilization range, and the second coefficient is greater than the first coefficient; respond to the current graphics processor utilization being in a third utilization range, use a third coefficient as a threshold adjustment coefficient corresponding to the initial decision threshold, wherein the lower limit of the third utilization range is greater than the upper limit of the second utilization range, and the third coefficient is greater than the second coefficient.

[0119] In some embodiments, the preset linear relationship is: ; in, This is the threshold adjustment coefficient. This represents the current graphics processor utilization rate.

[0120] In some embodiments, the inference module 300 is configured to: perform a cache reuse action or a full inference action based on the final decision threshold and the cumulative deviation.

[0121] In some embodiments, the inference module 300 is configured to: perform a cache reuse action in response to the cumulative deviation being less than the final decision threshold; and perform a full inference action in response to the cumulative deviation being greater than or equal to the final decision threshold.

[0122] In some embodiments, the inference module 300 is configured to: calculate the difference between the inference output data and the input data corresponding to the current time step; determine the input data for the next time step based on the inference output data corresponding to the current time step; and obtain the inference output data for the next time step based on the input data and the difference for the next time step.

[0123] In some embodiments, before performing a cache reuse action, the inference module 300 is configured to: identify whether the inference output data corresponding to the current time step meets the preset standard output conditions; if the inference output data corresponding to the current time step does not meet the preset standard output conditions, stop performing the cache reuse action and perform a full inference action.

[0124] In some embodiments, the inference module 300 is configured to: determine the model parameters and total number of time steps corresponding to the current inference process; determine the input data for the next time step based on the inference output data corresponding to the current time step; and output the inference output data for the next time step based on the input data for the next time step, the model parameters corresponding to the current inference process, and the total number of time steps.

[0125] In some embodiments, when the inference output data corresponding to the current time step does not meet the preset standard output conditions, and the cache reuse action is stopped, the inference module 300 is used to: generate a reminder message that the inference output data corresponding to the current time step does not meet the preset standard output conditions and the cache reuse action is stopped.

[0126] For a description of the features in the embodiment corresponding to the reasoning system, please refer to the relevant description of the embodiment corresponding to the reasoning method, which will not be repeated here.

[0127] Embodiments of the present invention also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above-described inference method embodiments.

[0128] Embodiments of the present invention also provide a computer-readable storage medium storing a computer program configured to execute the steps in any of the above-described inference method embodiments at runtime.

[0129] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0130] Embodiments of the present invention also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described inference method embodiments.

[0131] Embodiments of the present invention also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described inference method embodiments.

[0132] Any of the components, modules, units, parts, methods, and operations described herein can be implemented using software, firmware, hardware (e.g., fixed logic circuitry), manual processing, or any combination thereof. Alternatively or additionally, any functionality described herein can be performed at least in part by one or more hardware logic components, such as, but not limited to, a central processing unit (CPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), an application-specific standard product (ASSP), a system-on-a-chip (SoC), a complex programmable logic device (CPLD), a microprocessor (MCU), etc. The terms "system," "computing device," or "apparatus" as used herein encompass various means, devices, and machines for processing data, including, for example, one or more programmable processors, computers, SoCs, or combinations thereof. The apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or one or more combinations thereof. The aforementioned computer program (also known as a program, software, software application, app, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, object, or other unit suitable for a computing environment.

[0133] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0134] The foregoing has provided a detailed description of the reasoning method and electronic device provided by this invention. Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and core ideas of this invention. It should be noted that those skilled in the art can make various improvements and modifications to this invention without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this invention.

Claims

1. A reasoning method, characterized in that, The method is applied to a preset accelerator, wherein the method includes the following steps: Acquire the input data and inference output data within the target sliding window; Based on the input data and the inference output data, calculate the rate of change of data features within the target sliding window, the coefficient of variation of the rate of change of data features, and the cumulative deviation; An initial decision threshold is calculated based on the data feature change rate, the coefficient of variation, and the cumulative deviation of the data feature change. A final decision threshold is determined based on the initial decision threshold and the current graphics processor utilization. A cache reuse action or a full inference action is then performed based on the final decision threshold.

2. The method according to claim 1, characterized in that, The target sliding window includes at least one time step. Based on the input data and the inference output data, the rate of change of data features within the target sliding window is calculated, including: Based on the input data and the inference output data, calculate the input data deviation and inference output data deviation at least for a portion of the time steps within the target sliding window; The rate of change of data features within the target sliding window is obtained by the ratio of the inference output data deviation of at least a portion of the time steps to the input data deviation of the corresponding time steps.

3. The method according to claim 2, characterized in that, Based on the input data and the inference output data, the coefficient of variation of the rate of change of the data features within the target sliding window is calculated, including: Based on the rate of change of data features at least some time steps, calculate the mean and standard deviation of the rate of change of data features within the target sliding window at least some time steps; The coefficient of variation of the data feature change rate within the target sliding window is obtained by the ratio of the mean of the data feature change rate at least some time steps to the standard deviation of the data feature change rate at the corresponding time step.

4. The method according to claim 2, characterized in that, Based on the input data and the inference output data, the cumulative deviation of data feature changes within the target sliding window is calculated, including: Calculate the deviation of the input data at least for a portion of the time steps within the target sliding window; Calculate the product of the deviation of the input data at least a portion of the time steps within the target sliding window and the rate of change of the data characteristics at least a portion of the time steps within the target sliding window; The ratio of the product to the inference output data corresponding to the input data of at least some time steps is calculated to obtain the ratio corresponding to the at least some time steps. The cumulative deviation of data feature changes within the target sliding window is obtained by summing the ratios corresponding to at least some of the time steps.

5. The method according to any one of claims 1-4, characterized in that, The calculation of the initial decision threshold based on the data feature change rate, the coefficient of variation, and the cumulative deviation of the data feature changes includes: Calculate the coefficient of variation of the rate of change of the data features and the first product of the first scaling factor; Determine the instantaneous derivative of the cumulative deviation of the data feature changes, and calculate the second product of the instantaneous derivative of the cumulative deviation of the data feature changes and the second scaling factor; The initial decision threshold is calculated based on the first product value and the second product value.

6. The method according to claim 5, characterized in that, The step of determining the final decision threshold based on the initial decision threshold and the current graphics processor utilization includes: Determine the current utilization range of the graphics processor; Based on the current utilization range of the graphics processor, determine the threshold adjustment coefficient corresponding to the initial decision threshold; The final decision threshold is obtained by multiplying the threshold adjustment coefficient and the initial decision threshold.

7. The method according to claim 6, characterized in that, The step of determining the threshold adjustment coefficient corresponding to the initial decision threshold based on the current utilization range of the graphics processor includes: In response to the current graphics processor utilization being within a first utilization range, the first coefficient is used as the threshold adjustment coefficient corresponding to the initial decision threshold. In response to the current graphics processor utilization being in the second utilization range, a second coefficient is calculated based on the current graphics processor utilization according to a preset linear relationship, and the second coefficient is used as the threshold adjustment coefficient corresponding to the initial decision threshold, wherein the lower limit of the second utilization range is greater than the upper limit of the first utilization range, and the second coefficient is greater than the first coefficient. In response to the current graphics processor utilization being in the third utilization range, the third coefficient is used as the threshold adjustment coefficient corresponding to the initial decision threshold, wherein the lower limit of the third utilization range is greater than the upper limit of the second utilization range, and the third coefficient is greater than the second coefficient.

8. The method according to claim 7, characterized in that, The preset linear relationship is as follows: ； in, This is the threshold adjustment coefficient. This represents the current graphics processor utilization rate.

9. The method according to claim 1, characterized in that, The step of performing cache reuse or full inference based on the final decision threshold includes: Execute either cache reuse or full inference based on the final judgment threshold and the cumulative deviation of the data feature changes.

10. The method according to claim 9, characterized in that, The step of performing cache reuse or full inference based on the final decision threshold and the cumulative deviation of data feature changes includes: If the cumulative deviation of the data feature changes is less than the final decision threshold, a cache reuse action is performed. In response to the cumulative deviation of the data feature changes being greater than or equal to the final decision threshold, a full inference action is performed.

11. The method according to claim 10, characterized in that, The cache reuse action includes: Calculate the difference between the inference output data and the input data at the current time step; The input data for the next time step is determined based on the inference output data corresponding to the current time step; The inference output data for the next time step is obtained based on the input data for the next time step and the difference.

12. The method according to claim 11, characterized in that, Before performing cache reuse, the following is also included: Identify whether the inference output data corresponding to the current time step meets the preset standard output conditions; If the inference output data corresponding to the current time step does not meet the preset standard output conditions, the cache reuse action is stopped and a full inference action is performed.

13. The method according to claim 10, characterized in that, The execution of the full inference action includes: Determine the model parameters and total number of time steps corresponding to the current inference process; The input data for the next time step is determined based on the inference output data corresponding to the current time step; Based on the input data of the next time step, the model parameters corresponding to the current inference process, and the total number of time steps, the inference output data of the next time step is output.

14. The method according to claim 12, characterized in that, If the inference output data corresponding to the current time step does not meet the preset standard output conditions, the cache reuse action is stopped, and a full inference action is performed, including: A reminder message is sent indicating that the inference output data corresponding to the current time step does not meet the preset standard output conditions, and the cache reuse action is stopped.

15. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the reasoning method as described in any one of claims 1-14.

Images