An active thermal management method and product for end-side AI chips

Abstract

Embodiments of the present application provide an active thermal management method and product for an end-side AI chip, the active thermal management method comprising: obtaining a feature parameter of a current inference task running in an NPU of a chip to be regulated, wherein the feature parameter comprises: an input token number, a historical output token number, a queue depth of the current inference task, and an inference stage identifier; constructing a task feature vector according to at least the feature parameter, wherein the task feature vector comprises: an equivalent computing operation number, an equivalent memory data volume, a parallelism pressure factor, and the inference stage identifier; inputting the task feature vector into a target load prediction neural network model to determine a power consumption trajectory of the NPU in a plurality of future time windows; mapping the power consumption trajectory into a temperature change trajectory of the chip to be regulated according to an equivalent thermal model, and generating a regulation strategy for adjusting the chip to be regulated according to the temperature change trajectory.

Description

Technical Field

[0001] This application relates to the field of edge AI chip technology, and more specifically, the embodiments of this application relate to an active thermal management method and product for edge AI chips. Background Technology

[0002] Existing edge AI chips typically integrate a neural network processor (NPU) and are configured with a passive dynamic voltage-frequency scaling (DVFS) control architecture based on temperature thresholds. The regulation process based on this architecture involves the power management unit forcibly reducing the NPU's voltage and frequency to limit power consumption when the on-chip temperature sensor detects that the junction temperature exceeds a preset threshold. Once the temperature returns to normal, the original performance level is restored. This architecture's regulation action relies entirely on temperature feedback, resulting in a significant response lag from chip temperature rise to triggering regulation, and the regulation granularity is coarse. In high-dynamic load scenarios such as large-model inference on the edge, this lag often leads to repeated chip temperature overshoots and frequent forced frequency reductions, causing drastic fluctuations in inference latency and making it difficult to balance thermal safety and performance stability. Summary of the Invention

[0003] The purpose of this application is to provide an active thermal management method and product for edge AI chips. The embodiments of this application parse the characteristics of large-scale model inference tasks on the edge into computational and memory access load characteristics. A lightweight neural network is used to predict the future power consumption trajectory of the NPU, and then an equivalent thermal model is used to forward deduce the chip temperature change trend. Thus, before the actual load temperature rise occurs, differentiated and coordinated control is implemented on voltage, clustering frequency, fan speed, and inference accuracy. Some embodiments of this application also continuously correct the prediction and thermal model parameters through an online correction loop, forming a closed-loop adaptive control. Its technical effects include effectively reducing the chip's peak temperature, reducing the probability of overheating-induced forced frequency reduction, improving inference latency consistency, and enhancing the chip's thermal management stability under long-term operation and different environmental conditions.

[0004] In a first aspect, embodiments of this application provide an active thermal management method for an edge AI chip. The active thermal management method includes: acquiring characteristic parameters of a current inference task running in the NPU of the chip to be managed, wherein the characteristic parameters include: the number of input tokens, the number of historical output tokens, the queue depth of the current inference task, and an inference stage identifier; constructing a task feature vector based at least on the characteristic parameters, wherein the task feature vector includes: equivalent computational operands, equivalent memory access data volume, parallelism pressure factor, and the inference stage identifier, wherein the inference stage identifier is an enumerated variable used to indicate the stage of the current inference task; and the equivalent computational operands are used to characterize the busy level of the target computing resources of the NPU. The equivalent memory access data volume is an estimate of the total amount of data that the NPU needs to read and write from off-chip memory or on-chip cache when executing the current inference task; the parallelism pressure factor is a dimensionless normalized index used to characterize the amplification effect of the congestion of the current task queue of the NPU on power consumption; the task feature vector is input into the target load prediction neural network model to determine the power consumption trajectory of the NPU in multiple future time windows; the power consumption trajectory is mapped to the temperature change trajectory of the chip to be controlled according to the equivalent thermal model, and a control strategy for adjusting the chip to be controlled is generated according to the temperature change trajectory.

[0005] The embodiments of this application construct a task feature vector by parsing NPU inference task characteristics, predict future power consumption using a lightweight recurrent neural network, and then use an equivalent thermal model to infer chip temperature changes based on the predicted power consumption. This allows for proactive adjustment of the AI ​​model on the target side within a preset time window before a sudden load surge (e.g., multi-dimensional coordinated adjustment such as voltage pre-adjustment of the AI ​​chip, differentiated frequency adjustment of computation and memory access units, fan pre-acceleration, and accuracy pre-degradation). This technique effectively reduces chip peak temperature, lowers the probability of forced frequency reduction, and improves the consistency of inference latency, solving the time-scale mismatch problem between the dynamics of large-scale model load and thermal inertia on the target side.

[0006] In some embodiments, constructing the task feature vector based at least on the feature parameters includes: obtaining fixed architecture parameters of the edge-side large model running on the NPU, wherein the fixed architecture parameters include: number of layers, hidden dimension, and feedforward network intermediate dimension; if it is confirmed that the inference stage identifier indicates that the edge-side large model is in the Prefill stage, then the equivalent computational operands are determined using the following formula: O comp = N in × ( 2 × L × (4H² + 2H × D ff ) )× 10⁻ 9 , where N inLet L be the number of input tokens, H be the number of layers in the large edge model, and D be the hidden dimension of the large edge model. ff Let be the intermediate dimension of the feedforward network of the large edge model, 4H² be the computational cost of Q, K, and V projections and output projection in each attention mechanism layer, and 2H × Dff be the computational cost of the two fully connected layers in each feedforward network layer. If the inference stage identifier indicates that the large edge model is in the Decode stage, the equivalent computational operands are determined using the following formula: O comp = N outhist × [ 2 × L × (4H² + 2H × D ff ) + 2 × L × H × N in ×10⁻ 9 , where N outhist N represents the number of output tokens generated during the autoregressive decoding process. in Let L be the number of input tokens, H be the number of layers in the large edge model, and D be the hidden dimension of the large edge model. ff is the intermediate dimension of the feedforward network of the large end-side model.

[0007] The embodiments of this application accurately transform abstract inference task information into quantifiable equivalent computational operands through model fixed architecture parameters and stage differentiation formulas. This allows for a refined characterization of load differences between different input lengths and different inference stages, thereby providing highly relevant input features for subsequent power consumption prediction models, improving the accuracy of power consumption prediction, and providing a quantitative basis for implementing differentiated frequency adjustments in stages.

[0008] In some embodiments, constructing the task feature vector based at least on the feature parameters includes: Mdata = (N in +N outhist ) × L × H × 2 + (total model weights) × activation factor, where Mdata is the equivalent memory access data volume, the activation factor takes a first value in the Prefill stage and a second value in the Decode stage, and the second value is less than the first value.

[0009] The technical means of this application embodiment distinguishes the inference stage and introduces differentiated activation factors, and comprehensively quantifies the number of input and output tokens, model architecture parameters and deployment accuracy into an equivalent memory access data volume. It can accurately characterize the difference in memory access characteristics between the full weight loading in the Prefill stage and the KV cache access in the Decode stage, providing input features that are highly correlated with memory access power consumption for the subsequent power consumption prediction model, and providing a quantitative basis for implementing differentiated frequency adjustment of memory access units in stages.

[0010] In some embodiments, constructing a task feature vector based at least on the feature parameters includes: P = min(Q / Qmax, 1.0), where Q is the queue depth of the current inference task, Qmax is the preset maximum queue depth, and P is the parallelism pressure factor.

[0011] The technical means of this application's embodiments captures the congestion effect of multi-task parallelism on NPU hardware unit scheduling by determining the ratio of the current queue depth to the preset maximum queue depth, normalizes it to a dimensionless factor between zero and one, enabling the prediction model to perceive the additional peak power consumption caused by multi-task contention for computing resources, compensates for the parallel overhead that cannot be characterized by single-task operands and memory accesses alone, and improves the power consumption prediction accuracy in high-concurrency inference scenarios.

[0012] In some embodiments, the target load prediction neural network model includes an input layer, a GRU hidden layer, and a fully connected output layer. The input layer is used to receive the equivalent number of computational operands, the equivalent amount of memory access data, the parallelism pressure factor, and the inference stage identifier. The GRU hidden layer contains sixteen neurons and uses a gating mechanism to fuse the input with the historical hidden state and capture the temporal dependence of load dynamics. The output layer includes multiple neurons, and each neuron is used to output the predicted power consumption value at a future time.

[0013] The network architecture of the target prediction neural network model provided in the embodiments of this application combines the GRU gating mechanism with full integer quantization inference to achieve time-series prediction of future multi-step power consumption trajectories with extremely low chip area and power consumption overhead, and simultaneously outputs confidence scores for subsequent control decisions, solving the problem that traditional prediction models cannot be deployed on edge control processors due to high floating-point computation overhead.

[0014] In some embodiments, the active thermal management method further includes: initiating the inference task of the edge-side large model on the chip to be controlled, constructing multiple inference loads, wherein the multiple inference loads cover various operating scenarios from the Prefill stage to the Decode stage by adjusting the values ​​of controllable parameters of the inference task, and the types of controllable parameters include: input prompt length, output generation length, and number of concurrent tasks; before each inference task is issued, reading the number of input tokens, the number of historical output tokens, the queue depth, and the inference stage identifier corresponding to the current inference task from the task scheduler to obtain a sample four-dimensional feature vector, which is used as the input feature of a sample and a timestamp is recorded; starting from the recorded timestamp, using a power sensor to continuously collect the average power consumption value of the NPU in multiple subsequent time windows at preset intervals, and using the average power consumption value as the label value of this sample; binding the sample four-dimensional feature vector and the label value into a training sample and storing it in the dataset; training the load prediction neural network model according to the dataset to obtain the target load prediction neural network model.

[0015] The training method provided in some embodiments of this application runs a large edge model on the chip to be controlled and constructs diverse loads covering different input lengths, output lengths and concurrency levels. It directly collects the real power consumption of multiple future time windows from the on-chip power sensor as labeled data, so that the training data can fully reflect the dynamic power consumption characteristics of the chip to be controlled in the actual inference scenario. This trains a high-precision load prediction neural network model with strong generalization ability, effectively avoiding the prediction deviation caused by the inconsistency between traditional offline modeling and actual operating conditions.

[0016] In some embodiments, training the load prediction neural network model based on the dataset to obtain the target load prediction neural network model includes: randomly selecting a batch of samples from the dataset; inputting the four-dimensional feature vectors of the batch of samples into the input layer to pass the four-dimensional feature vectors corresponding to the batch of samples to the GRU hidden layer; the GRU hidden layer calculating the gating value and updating the hidden state based on the current weights and the hidden state at the previous time step; the output layer generating multiple predicted power consumption values ​​and a confidence value; determining the total loss based on the predicted power consumption values, the labeled values, and the loss function, wherein the loss function consists of the sum of a main loss and an auxiliary loss, the main loss being the mean square error between multiple predicted power consumption values ​​and multiple labeled values ​​of real power consumption, and the auxiliary loss being the confidence calibration loss; and adjusting the training parameters based on the total loss until the training is completed to obtain the target load prediction neural network model.

[0017] The training method provided in the embodiments of this application collects power consumption data of real inference load directly on the chip to be controlled as annotation, and designs a dual loss function including power consumption mean square error and confidence calibration. This enables the load prediction neural network model to not only accurately output multi-step power consumption trajectory, but also to self-evaluate the reliability of its own prediction, providing high-quality information with uncertainty measurement for subsequent thermal regulation decisions, and improving the safety margin control capability of active regulation.

[0018] In some embodiments, mapping the power consumption trajectory to the temperature change trajectory of the chip to be controlled according to the equivalent thermal model, and generating a control strategy for adjusting the chip to be controlled based on the temperature change trajectory, includes: taking the current junction temperature of the chip to be controlled as the initial value, sequentially substituting the predicted power consumption values ​​of each time window in the power consumption trajectory into the equivalent thermal model, recursively calculating the predicted temperature values ​​of the chip to be controlled in each time window, and connecting the predicted temperature values ​​to obtain the temperature change trajectory; determining the temperature risk level based on the peak temperature and temperature rise rate in the temperature change trajectory; and generating temperature adjustment strategies corresponding to different inference stages based at least on the temperature risk level. The model parameters of the equivalent thermal model include equivalent thermal resistance and equivalent thermal capacity, which are updated online using a recursive least squares method based on the deviation between the actual temperature and the predicted temperature through an online correction loop.

[0019] The embodiments of this application provide a method for dynamically updating the parameters of an equivalent thermal model and a chip control strategy adapted to the corresponding inference stage based on the future temperature predicted by the equivalent thermal model and the inference stage, thereby improving the accuracy of the temperature predicted by the equivalent thermal model and enhancing the effect of active chip control.

[0020] Secondly, some embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement the active thermal management method as described in any of the embodiments included in the first aspect.

[0021] Thirdly, some embodiments of this application provide an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it can implement the active thermal management method as described in any of the embodiments included in the first aspect. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is an architecture diagram of an active thermal management system for an edge AI chip provided in an embodiment of this application.

[0024] Figure 2 This is a flowchart illustrating an active thermal management method for an edge AI chip according to an embodiment of this application.

[0025] Figure 3 This application provides a schematic diagram of the electronic device composition. Detailed Implementation

[0026] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0027] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0028] With the rapid development of edge AI applications, the demand for lightweight, large-scale models such as Gemma and Qwen, which can be deployed on edge, continues to grow in inference applications on mobile terminals, edge computing devices, and embedded systems. The inventors of this application discovered in their research that such models typically exhibit distinct phased load characteristics during inference. Specifically, the Prefill phase is computationally intensive with a rapid increase in power consumption, while the Decode phase is memory-intensive with significant power fluctuations, leading to rapid changes in chip temperature within a short period.

[0029] Due to limitations in size, heat dissipation structure, and power supply conditions, edge devices in related technologies are typically difficult to configure with strong active cooling devices. Chip thermal management mainly relies on passive cooling and temperature feedback-based control mechanisms. Existing thermal management methods generally suffer from problems such as response lag, coarse adjustment granularity, significant temperature overshoot, and large performance fluctuations. Especially under continuous inference loads, they are prone to triggering thermal throttling, causing significant deterioration of inference latency. In existing technologies, threshold-based passive DVFS control schemes only adjust voltage and frequency after the temperature exceeds the limit, making it difficult to respond promptly to microsecond-level load changes during large-scale model inference on the edge.

[0030] The inventors of this application discovered in their research that while machine learning-based load prediction schemes provided by related technologies can detect load trends in advance to some extent, they are typically not coupled with thermal models and lack the ability to synchronously extrapolate temperature evolution trajectories, thus failing to be further transformed into reasonable control strategies. The inventors of this application also found that some temperature prediction schemes based on thermal models in related technologies often rely on offline calibration parameters, making it difficult to adapt to chip aging, packaging changes, and environmental temperature drift, and failing to form a closed-loop active control system oriented towards inference tasks. Therefore, related technologies have not effectively solved the time-scale mismatch problem between load dynamics and thermal inertia in edge-side large-model inference scenarios, making temperature overshoot and performance cliff phenomena difficult to avoid.

[0031] To address the aforementioned technical issues, some embodiments of this application parse the characteristics of large-scale model inference tasks on the edge into computational and memory access load characteristics. A lightweight neural network is used to predict the future power consumption trajectory of the NPU. Then, an equivalent thermal model is used to forward deduce the chip temperature change trend based on the predicted future power consumption. This proactively implements differentiated and coordinated control of voltage, clustering frequency, fan speed, and inference accuracy before the actual load temperature rise occurs. Some embodiments of this application also continuously correct the prediction and thermal model parameters through an online correction loop, forming a closed-loop adaptive control. The technical effects include: effectively reducing the chip's peak temperature, reducing the probability of overheating-induced forced frequency reduction, improving inference latency consistency, and enhancing the chip's thermal management stability under long-term operation and different environmental conditions.

[0032] Before introducing the embodiments of this application, the meaning of the relevant terms in this application will be briefly explained first.

[0033] NPU: Neural Processing Unit, is a hardware module specifically designed to accelerate neural network computation. It is usually integrated into edge AI chips and is used to efficiently perform matrix operations, convolution, and other operations in inference tasks.

[0034] In neural network inference tasks, a token is the smallest processing unit after text or sequence data has been segmented. For large language models, the input tokens typically correspond to a vector representation of a word or character, and their number directly affects the computational load and memory access requirements of the NPU.

[0035] Equivalent computational operands: A derived metric used to estimate the effective computational load required for neural network inference tasks to be executed on an NPU. Equivalent computational operands are not simply the number of multiplication and addition operations, but rather, considering the characteristics of the NPU's computing architecture, they uniformly convert the computational intensity of different operators into equivalent scalar operands to reflect the actual workload of computing resources.

[0036] Equivalent Memory Access Data Volume: A derived characteristic metric used to estimate the total amount of data read from and written to on-chip cache and off-chip memory during NPU execution of inference tasks. This metric equivalently calculates the bandwidth and latency characteristics of different levels of storage access, quantifying the pressure of the task on the storage subsystem.

[0037] Parallelism pressure factor: A normalized derivative metric used to characterize the degree of competition for NPU computing resources by inference tasks waiting to be executed in the task queue. The greater the queue depth, the higher the parallelism pressure factor, meaning that multiple tasks are competing for computing units simultaneously, which may lead to amplified power consumption due to resource conflicts. This factor is used to correct power consumption predictions.

[0038] Inference Phase Identifier: An enumerated state variable used to specify the current phase of the inference task. For Transformer-type models, there are typically two phases: Prefill and Decode. The computational and memory access characteristics of these two phases differ significantly, therefore, they need to be identified for separate modeling.

[0039] Power consumption trajectory: Sequence data showing the change of NPU power consumption over time over a future period, output by a neural network prediction model. The power consumption trajectory maps the task feature vector to predicted power consumption values ​​for multiple consecutive time windows, forming a power curve used to predict upcoming power hotspots.

[0040] Thermal model: An equivalent mathematical model or simulation tool that maps power consumption information to the trajectory of chip temperature changes. The thermal model comprehensively considers the spatial distribution of the chip package's thermal resistance, thermal capacity, ambient temperature, and power consumption. It can calculate the future trend of NPU core temperature changes based on the power consumption trajectory, serving as a basis for control decisions.

[0041] Prefill: The prefill stage is the first stage of Transformer inference. This stage processes the entire input cue sequence in parallel at once, calculating the key-value pair cache at all positions. It is characterized by computational intensity, numerous large matrix multiplications, high power consumption, but short time.

[0042] Decode: The decoding stage is the second stage of Transformer inference. This stage generates output autoregressively per token, generating only one token per inference iteration, and using key-value pairs cached in previous stages. Typical characteristics include memory-intensive operation, low computational intensity but repeated execution, and long duration.

[0043] Please refer to Figure 1 , Figure 1 An active thermal management system for an edge AI chip, provided in some embodiments of this application, includes:

[0044] The NPU task scheduler 101 is hardware deployed in the chip to be controlled, used to manage the inference task queue and distribute task instructions to the NPU computing cluster. In this system, the NPU task scheduler is also configured with a feature parameter output interface, used to send the number of input tokens, the number of historical output tokens, the task queue depth, and the inference stage identifier of the current inference task to the load prediction engine 102 in the form of digital signals. A token is an input / output data unit for an inference task.

[0045] The load prediction engine is a newly added unit in this embodiment of the invention. This unit contains three sub-components: a GRU network inference unit, an INT4 weight register file, and an input quantizer. Figure 1 (Not shown). The GRU network inference unit consists of an INT4 multiply-accumulate array, a piecewise linear lookup table, and a hidden state register file, and is responsible for performing forward inference of the lightweight gated recurrent network. The INT4 weight register file stores the network weight parameters after offline training and quantization. The input quantizer is responsible for converting the received floating-point feature vectors into INT4 format. The output of the load prediction engine generates two signals: one is a sequence of power consumption trajectories for multiple future time windows, which is sent to the thermal trajectory extrapolator 107; the other is the prediction confidence value, which is simultaneously sent to the thermal trajectory extrapolator and the online correction loop 106.

[0046] The thermal trajectory extrapolator 107 is a newly added unit in some embodiments of the present invention, which internally includes three sub-components: a first-order RC recursive calculation circuit, a thermal parameter register, and a threshold comparison logic. Figure 1 (Not shown). The first-order RC recursive calculation circuit consists of a multiplier, an adder, and an accumulator register, responsible for performing temperature recursive calculations. The thermal parameter register stores the current values ​​of thermal resistance and thermal capacity. The threshold comparison logic consists of a digital comparator array and a threshold register, responsible for determining the temperature risk level. This module has four inputs, receiving the power consumption trajectory sequence from the load prediction engine, the current chip junction temperature from the on-chip temperature sensor 103, and so on. Figure 1 The ambient temperature (not shown) comes from the board-level sensor 105, and the updated thermal parameters come from the online correction loop 106. The output of the thermal trajectory extrapolator generates a pre-control trigger signal containing three state codes: no control, regular pre-control, and emergency pre-control, which is sent to the active coordinated control actuator. The online correction loop 106 is also used to receive the measured power consumption sequence collected by the power consumption sensor 104, and to compare these power consumption values ​​with the predicted power consumption values ​​of the load prediction engine to generate an updated bias for parameter readjustment of the target load prediction neural network model in the load prediction engine.

[0047] The active coordinated control actuator internally includes multiple parallel-arranged control sub-modules, including: a voltage pre-regulator 108, which internally contains a programmable voltage recognition register and a digital-to-analog converter, outputting an analog reference voltage to an external power management unit; a non-uniform frequency regulator 109, which internally contains two independently configurable clock dividers and a glitch-free clock gating unit, generating independent clock signals for the NPU computing cluster and memory access cluster, respectively; and a fan pre-acceleration driver 110, which internally contains a PWM duty cycle register and a PWM waveform generator, outputting a PWM square wave to the chip's cooling fan drive pin. In some other embodiments of this application, the active coordinated control actuator further includes a precision pre-degradation buffer (…). Figure 1 (Not shown), this precision pre-degradation buffer contains a precision control status register and outputs a precision mode switching signal to the data path selector of the NPU computing array. The inputs of all submodules of the active coordinated control actuator are connected to the pre-control trigger signal of the thermal trajectory extrapolator, responding in parallel within the same control cycle. For example, the output signal of the voltage regulator is connected to... Figure 1 The power management unit (PMU) 114 controls the NPU computing cluster 112 and the NPU access cluster 113 respectively; the non-uniform frequency regulator outputs the computing cluster clock to the NPU computing cluster and the memory access cluster clock to the NPU access cluster respectively; the fan pre-acceleration driver 110 is used to output PWM signals to the cooling fan 115.

[0048] The online correction loop internally includes an error calculation circuit, a recursive least squares parameter update state machine, and parameter write-back control logic. Figure 1 (Not shown). The error calculation circuit consists of a subtractor and an absolute value accumulator. The input of the online correction loop receives four signals: the actual temperature sequence from the on-chip temperature sensor, the actual power consumption sequence from the power consumption sensor, the predicted power consumption trajectory from the load prediction engine, and the predicted temperature trajectory from the thermal trajectory inferrer. The output of the online correction loop generates two update signals: one is to write the updated thermal resistance and thermal capacity values ​​into the thermal parameter register of the thermal trajectory inferrer, and the other is to write the incremental correction value of the output layer bias of the prediction network into the weight register file of the load prediction engine. In some embodiments of this application, the update operation is performed only during the NPU inference idle period to avoid interfering with real-time inference.

[0049] The NPU compute cluster 112 and NPU memory cluster 113 are the hardware components deployed on the chip to be controlled, serving as the core computing resources for performing large-scale model inference tasks. The NPU compute cluster receives the compute cluster clock signal output by the non-uniform frequency regulator, and the NPU memory cluster receives the memory cluster clock signal output by the non-uniform frequency regulator; both operate at independent frequencies. Their power supply voltage is uniformly provided by the external power management unit based on the reference voltage output by the voltage pre-regulator.

[0050] Figure 1 The on-chip temperature sensor is integrated inside the chip to be controlled, which collects the chip junction temperature in real time and outputs it to the thermal trajectory extrapolator and the online correction loop respectively. Figure 1 The power consumption sensor is integrated into the chip's power domain to collect the dynamic power consumption of the NPU subsystem in real time and output it to the online correction loop. Figure 1 The board-level sensors are arranged on the circuit board to collect ambient temperature and output it to the thermal trajectory extrapolator.

[0051] The following is combined with Figure 2 This application provides exemplary embodiments of an active thermal management method for edge AI chips, which includes:

[0052] S210: Obtain the feature parameters of the current inference task running in the NPU of the chip to be controlled.

[0053] It should be noted that, in the embodiments of this application, the feature parameters described in S210 include: the number of input tokens, the number of historical output tokens, the queue depth of the current inference task, and the inference stage identifier.

[0054] S220, at least construct a task feature vector based on the feature parameters.

[0055] It should be noted that the task feature vector mentioned in S220 includes: equivalent computational operands, equivalent memory access data volume, parallelism pressure factor, and the inference stage identifier. The inference stage identifier is an enumerated variable used to indicate the stage of the current inference task. The equivalent computational operands are used to characterize the busyness of the target computing resources (such as multiply-accumulate arrays) of the NPU. The equivalent memory access data volume is an estimate of the total amount of data that the NPU needs to read and write from off-chip memory or on-chip cache when executing the current inference task. The parallelism pressure factor is a dimensionless normalized index used to characterize the amplification effect of the congestion of the current task queue of the NPU on power consumption.

[0056] For example, in some embodiments of this application, the task feature vector is determined by the following strategy: the equivalent computational operands, the equivalent memory access data volume, the parallelism pressure factor, and the inference stage identifier constitute a four-dimensional original feature vector; the target four-dimensional feature vector (i.e., the task feature vector) is obtained by normalizing the original thought feature vector.

[0057] S230, the task feature vector is input into the target load prediction neural network model to determine the power consumption trajectory of the NPU in multiple future time windows.

[0058] It should be noted that, in the embodiments of this application, the fully connected output layer of the target load prediction neural network model has multiple main output neurons, each corresponding to the predicted power consumption value for a future time window. The multiple neurons output the average NPU power consumption value for the first to Nth time windows in the future, in milliwatts, according to the order of the time windows. Arranging these N discrete predicted power consumption values ​​in chronological order forms a power consumption change sequence within multiple consecutive time windows, and connecting this sequence yields a power consumption trajectory.

[0059] In the embodiments of this application, the target load prediction neural network model and the load prediction neural network model have the same architecture. The difference lies in that the target load prediction neural network model is trained using labeled data to obtain the load prediction neural network model. In some embodiments of this application, the target load prediction neural network model also adjusts its model parameters based on the output of the online correction loop during the online application phase.

[0060] S240, the power consumption trajectory is mapped to the temperature change trajectory of the chip to be controlled according to the equivalent thermal model, and a control strategy for adjusting the chip to be controlled is generated according to the temperature change trajectory.

[0061] The following example illustrates... Figure 2 How each step is implemented.

[0062] For example, in some embodiments of this application, the step of constructing a task feature vector at least based on the feature parameters in step S210 includes:

[0063] The first step is to obtain the fixed architecture parameters of the large edge model running on the NPU, wherein the fixed architecture parameters include: number of layers, hidden dimension, and intermediate dimension of the feedforward network.

[0064] For example, in some embodiments of this application, the values ​​of the fixed architecture parameters are determined as follows: During the chip deployment phase, the architecture parameters of the large edge model (such as Gemma-2B) supported by the NPU are pre-written into the chip's non-volatile memory or firmware configuration table. When the system starts and loads the model, the control processor reads three constant values—the number of layers, the hidden dimension, and the intermediate dimension of the feedforward network—from this storage area and writes them into the corresponding registers for computation. Taking Gemma-2B as an example, the number of layers is 18, the hidden dimension is 2048, and the intermediate dimension of the feedforward network is 16384. These values ​​are directly derived from the publicly released technical report of the model and are inherent properties of the model, not changing with the inference task.

[0065] The second step is to obtain the values ​​of the equivalent computational operands corresponding to different inference stages.

[0066] If it is confirmed that the inference stage identifier indicates that the end-side large model is in the Prefill stage, then the equivalent computational operands are determined using the following formula: O comp = N in × ( 2 × L × (4H² + 2H × D ff ) ) × 10⁻ 9 , where N in Let L be the number of input tokens, H be the number of layers in the large edge model, and D be the hidden dimension of the large edge model. ff 4H² represents the intermediate dimension of the feedforward network in the large end-side model, and 4H² represents the computational cost of Q, K, and V projections and the output projection in each attention mechanism layer; 2H × D ff This represents the computational cost of two fully connected layers in each feedforward network.

[0067] If it is confirmed that the inference stage identifier indicates that the end-side large model is in the Decode stage, then the equivalent computational operands are determined using the following formula: O comp = N outhist × [ 2 × L × (4H² + 2H × D ff ) + 2 × L × H × N in ] × 10⁻ 9 , where N outhist N represents the number of output tokens generated during the autoregressive decoding process. in Let L be the number of input tokens, H be the number of layers in the large edge model, and D be the hidden dimension of the large edge model. ff Let N be the intermediate dimension of the feedforward network of the large-scale end-side model. An embodiment corresponding to this formula uses the number of input tokens N. in As an approximate cardinality for the length of the attention history, it is suitable for scenarios where the input length is much larger than the output length.

[0068] It should be noted that, in some other embodiments of this application, if the inference stage indicates that the large model on the representation side is in the Decode stage and higher accuracy in estimating computational operands is required, the equivalent computational operands are calculated using the following formula: O comp =N outhist × [ 2 × L × (4H² + 2H × D ff ) + 2 × L × H × (N in + N outhist ) ] × 10⁻ 9 , where N outhist N represents the number of output tokens generated during the autoregressive decoding process. inLet L be the number of input tokens, H be the number of layers in the large edge model, and D be the hidden dimension of the large edge model. ff N represents the intermediate dimension of the feedforward network of the large end-side model. in + N outhist The total history length corresponding to each generation step. This embodiment is suitable for scenarios where the input length is similar to the output length or the output length significantly exceeds the input length, and can avoid underestimating the attention computation cost due to ignoring generated tokens.

[0069] In other embodiments of this application, if it is necessary to perform token-by-token precise accumulation of the attention computation cost for each generation step in the Decode stage, the equivalent computational operands are calculated as follows: The specific meanings of the relevant parameters in this formula can be found above. k represents the k-th generation step, and the total history length at the k-th step is... + k - 1.

[0070] This embodiment is suitable for scenarios that require extremely high accuracy in power consumption prediction and can tolerate a small amount of additional computational overhead.

[0071] It should be noted that the following parameters L, H, and D... ff These are the architectural constants of the large edge model itself. For example, in Gemma-2B, L is 18, H is 2048, and D is... ff It is 16384, which can be obtained directly from the model technical report or configuration file. Nin and Nothist, however, change in real time and originate from the NPU task scheduler.

[0072] In some embodiments of this application, the step of constructing a task feature vector based at least on the feature parameters includes the following formula for determining the equivalent memory access data volume: Mdata = (N in +N outhist ) × L × H × 2 + (total model weights) × activation factor, where Mdata is the equivalent memory access data volume, the activation factor takes a first value in the Prefill stage and a second value in the Decode stage, and the second value is less than the first value.

[0073] For example, in some embodiments of this application, the equivalent memory access data volume Mdata is the sum of a first value and a second value. The first value is the sum of the number of input tokens and the number of historical output tokens multiplied by the number of layers, the hidden dimension, and the number 2. The second value is the constant of the total model weights multiplied by the activation factor. The multiplication of 2 in the first value indicates that FP16 half-precision storage is used, and each parameter occupies 2 bytes. The activation factor is 1.0 in the Prefill stage because all weights need to be fully loaded from memory in this stage. It is 0.1 in the Decode stage because the weight reuse rate is high in this stage, and the memory access pressure is mainly concentrated on the read and write of the KV cache.

[0074] In some embodiments of this application, the determination of the parameters involved in the above formula includes: the number of input tokens is directly read from the current task descriptor issued by the NPU task scheduler; the number of historical output tokens is read from the NPU autoregressive decoding status register, which increments after each token is generated during the Decode phase; the number of layers and hidden dimensions are read from the model fixed architecture parameter table pre-stored in the firmware of the chip to be controlled, using the same set of constants as when calculating equivalent computational operands; the total model weight is a pre-calculated and stored constant. For example, it is calculated based on the number of parameters and deployment precision of the large model on the running edge. For example, Gemma-2B has approximately 2 billion parameters, and if deployed with INT8 precision, the total model weight is approximately 2GB. This value is written into the firmware configuration table during model deployment for direct use during calculation; the activation factor is determined based on the inference phase identifier. If the identifier is the Prefill phase, the activation factor is 1.0; if the identifier is the Decode phase, the activation factor is 0.1. The values ​​are based on the following: during the Prefill stage, all weights need to be fully loaded from off-chip memory once; during the Decode stage, the weight reuse rate is extremely high, with only a small number of layers and caches being accessed frequently, and the memory access pressure is mainly concentrated on KV cache read and write.

[0075] In some embodiments of this application, the step of constructing a task feature vector based at least on the feature parameters includes the following formula for determining the parallelism pressure factor: P = min(Q / Qmax, 1.0), where Q is the queue depth of the current inference task, Qmax is the preset maximum queue depth, and P is the parallelism pressure factor.

[0076] Understandably, the min function in the formula for calculating the parallel pressure factor P truncates the calculation result between 0 and 1, ensuring that the factor is dimensionless and normalized.

[0077] It should be noted that the calculation formula for the parallelism pressure factor involves the following parameters: The queue depth Q of the current inference task is used to characterize the total number of inference tasks waiting to be executed in the NPU task scheduler at the current moment. This queue depth Q is obtained by directly reading it from the hardware status register of the NPU task scheduler. This register is incremented by one when each task enters the queue and decremented by one when a task is scheduled for execution by the NPU and removed from the queue, reflecting the current number of tasks to be processed in real time. The preset maximum queue depth Qmax is used to characterize a pre-set baseline value, representing the maximum depth of the NPU task queue under the target high load state, and is used as a normalization reference. The method for determining the maximum queue depth includes: This value is determined based on actual chip operation testing. For example, during the system deployment phase, the NPU is tested under different concurrent inference pressures, the distribution of task queue depths is statistically analyzed, and the depth value covering more than 95% of the working scenarios is selected as Qmax. For example, for a typical lightweight large model inference scenario on a mobile device, this value can be preset to 8. Qmax is written as a system tuning parameter into the chip firmware configuration table and read from the configuration table when constructing task feature vectors. The parallelism pressure factor P is a dimensionless index between zero and one, used to characterize the amplification effect of the current task queue congestion on power consumption in the NPU. When the queue depth exceeds Qmax, the factor is 1.0, indicating a high parallelism pressure state; when the queue is empty, the factor is 0, indicating no parallelism pressure.

[0078] In some embodiments of this application, the target load prediction neural network model includes an input layer, a GRU hidden layer, and a fully connected output layer. The input layer is used to receive the equivalent computational operands, the equivalent memory access data volume, the parallelism pressure factor, and the inference stage identifier. The GRU hidden layer contains sixteen neurons, and the GRU hidden layer fuses the input with the historical hidden state through a gating mechanism to capture the temporal dependence of load dynamics. The output layer includes multiple neurons, and each neuron is used to output the predicted power consumption value at a future time.

[0079] For example, in some embodiments of this application, the target load prediction neural network model consists of an input layer, a GRU hidden layer, and a fully connected output layer. The input layer receives a four-dimensional floating-point feature vector composed of the equivalent computation operands, the equivalent memory access data volume, the parallelism pressure factor, and the inference stage identifier, and independently quantizes each dimension into INT4 format according to a pre-stored quantization step size, outputting a four-dimensional integer vector to the GRU hidden layer.

[0080] The GRU hidden layer contains sixteen neurons and has three computational pathways: a reset gate, an update gate, and candidate hidden states. Each pathway contains a 4×16 input weight matrix and a 16×16 state weight matrix. All weights are stored in INT4 format, and the biases are in INT8 format. The GRU hidden layer receives the INT4 output of the input layer and the INT8 hidden state vector from the previous time step. It performs gating computation using a Sigmoid and Tanh activation function implemented with integer multiplication and accumulation and a piecewise linear lookup table, outputting a sixteen-dimensional INT8 hidden state vector.

[0081] The fully connected output layer consists of a main output branch and an auxiliary output branch. The main output branch contains a 16×8 INT4 weight matrix and an eight-dimensional bias. After linear transformation and inverse quantization of the hidden state vector, it outputs the predicted power consumption values ​​for eight future time windows, each with a length of 50 milliseconds, in milliwatts. The auxiliary output branch contains a 16×1 INT4 weight matrix and a one-dimensional bias. After Sigmoid activation, it outputs the prediction confidence.

[0082] In some embodiments of this application, the active thermal management method further includes the following training method:

[0083] The first step is to start the inference task of the large-scale model on the chip to be controlled, and construct multiple inference loads. The multiple inference loads cover various running scenarios from the Prefill stage to the Decode stage by adjusting the values ​​of the controllable parameters of the inference task. The types of controllable parameters include: input prompt length, output generation length, and number of concurrent tasks.

[0084] The second step is to read the number of input tokens, the number of historical output tokens, the queue depth, and the inference stage identifier corresponding to the current inference task from the task scheduler to obtain a four-dimensional feature vector of the sample, which is used as the input feature of a sample and the timestamp is recorded.

[0085] The third step involves using the recorded timestamp as a starting point, employing a power consumption sensor to continuously collect the average power consumption value of the NPU over multiple subsequent time windows at preset intervals, and using the average power consumption value as the label value for this sample.

[0086] The fourth step is to bind the four-dimensional feature vector of the sample and the labeled value into a training sample and store it in the dataset.

[0087] The fifth step is to train the load prediction neural network model based on the dataset to obtain the target load prediction neural network model.

[0088] For example, in some embodiments of this application, the fifth step of training the load prediction neural network model based on the dataset to obtain the target load prediction neural network model includes:

[0089] The first sub-step involves randomly selecting a batch of samples from the dataset.

[0090] For example, the sub-step includes: randomly selecting a batch of samples from the dataset, each batch containing thirty-two samples.

[0091] The second sub-step involves inputting the four-dimensional feature vectors of the batch of samples into the input layer to pass each four-dimensional feature vector corresponding to the batch of samples to the GRU hidden layer; the GRU hidden layer calculates the gating value based on the current weights and the hidden state at the previous time step and updates the hidden state; the output layer generates multiple predicted power consumption values ​​and a confidence value.

[0092] For example, the second sub-step includes: inputting the four-dimensional feature vector of the batch of samples into the input layer, and passing it from the input layer to the GRU hidden layer; the GRU hidden layer calculates the gating value based on the current weights and the hidden state at the previous time step and updates the hidden state; the output layer generates eight predicted power values ​​and one confidence value.

[0093] The third sub-step involves determining the total loss based on the predicted power consumption values, the labeled values, and the loss function. The loss function consists of the sum of a main loss and an auxiliary loss. The main loss is the mean square error between multiple predicted power consumption values ​​and multiple labeled values ​​of actual power consumption. The auxiliary loss is the confidence calibration loss. For example, the meaning of the calibration loss is explained below using eight predicted values ​​as an example. The calculation method is as follows: First, calculate the average absolute percentage error between the eight predicted values ​​and the eight labeled values. If this error exceeds a preset threshold, such as 15%, the expected confidence level should be close to zero. If the error is below the threshold, the expected confidence level should be close to one.

[0094] The fourth sub-step involves adjusting the training parameters based on the total loss until the training is completed and the target load prediction neural network model is obtained.

[0095] It should be noted that, for the training process, if the Adam optimizer is used, the network weights and biases are updated via backpropagation based on the total loss with an initial learning rate of 0.001. The embodiments of this application require repeating the first to fourth sub-steps to traverse all training data, training for a total of thirty epochs until the loss converges, obtaining the network after full-precision training. The network after full-precision training is then retrained using quantization-aware retraining. In the forward propagation, the approximate errors of INT4 weight quantization and piecewise linear lookup table activation are simulated. A pass-through estimator is used for backpropagation, updating the floating-point shadow weights with a learning rate of 0.0001. Training continues until the loss converges again, deriving the INT4 quantized weights and scaling factor, thus obtaining the target load prediction neural network model.

[0096] In some embodiments of this application, step S240, which maps the power consumption trajectory to the temperature change trajectory of the chip to be regulated based on an equivalent thermal model, and generates a regulation strategy for adjusting the chip to be regulated based on the temperature change trajectory, includes:

[0097] The first step is to take the current junction temperature of the chip to be controlled as the initial value, and then substitute the predicted power consumption values ​​of each time window in the power consumption trajectory into the equivalent thermal model in sequence to recursively calculate the predicted temperature values ​​of the chip to be controlled in each time window. Finally, the predicted temperature values ​​are connected to obtain the temperature change trajectory.

[0098] For example, the first step includes:

[0099] The current values ​​of equivalent thermal resistance and equivalent thermal capacity are read from the firmware configuration table of the chip to be controlled, the current chip junction temperature is read, and the ambient temperature is read from the board-level sensor. The eight predicted power consumption values ​​output by the target load prediction neural network model are used as the input power consumption of the thermal model in each future time step, in the order of 50 milliseconds per time window. Starting from the current chip junction temperature as the initial temperature, each predicted power consumption value is substituted into the discrete recursive formula of the first-order RC equivalent thermal model in turn, starting from the first time window, to calculate the chip predicted temperature at the corresponding time step by step. A total of eight time steps are performed to obtain a temperature change trajectory covering the next 400 milliseconds.

[0100] The second step is to determine the temperature risk level based on the peak temperature and the rate of temperature rise in the temperature change trajectory.

[0101] For example, the second step includes: extracting the predicted peak temperature from the temperature change trajectory, calculating the difference between the predicted peak temperature and the current chip junction temperature, dividing by the total prediction time to obtain the predicted temperature rise rate, and determining based on a preset temperature threshold: if the predicted peak temperature exceeds a second temperature threshold and the confidence level is greater than 0.6, it is determined to be an emergency risk. If the predicted peak temperature is between the first and second temperature thresholds and the predicted temperature rise rate exceeds 2 degrees Celsius per second, it is determined to be a normal risk. All other cases are determined to be no risk.

[0102] The third step is to generate temperature adjustment strategies corresponding to different inference stages, based at least on the temperature risk level.

[0103] For example, the third step includes: when an emergency risk is identified, generating a first type of adjustment strategy: sending a voltage boost command to the power management unit to increase the NPU's supply voltage from the current value to a preset higher voltage level; sending a maximum duty cycle command to the fan driver; sending a precision degradation command to the NPU precision control port to force subsequent inference to switch to INT4 mode; simultaneously, reading the current inference stage identifier, if in the Prefill stage, setting the compute cluster clock to full frequency and the memory cluster clock to reduced frequency, and if in the Decode stage, setting the memory cluster clock to full frequency and the compute cluster clock to reduced frequency. When a normal risk is identified, generating a second type of adjustment strategy: reading the current inference stage identifier, if in the Prefill stage, setting the compute cluster clock to full frequency and the memory cluster clock to reduced frequency; if in the Decode stage, setting the memory cluster clock to full frequency and the compute cluster clock to reduced frequency; simultaneously sending a duty cycle boost command to the fan driver to a preset higher value and maintaining it for at least one second. When the risk is determined to be no, the current voltage, frequency, and fan speed are maintained unchanged; each instruction in the generated adjustment strategy is sent to the corresponding execution unit in parallel within the same control cycle to ensure that it takes effect within the preset time window before the load surge.

[0104] It should be noted that, in some embodiments of this application, the model parameters of the equivalent thermal model include equivalent thermal resistance and equivalent heat capacity. The equivalent thermal resistance and equivalent heat capacity are updated online through an online correction loop, based on the deviation between the actual temperature and the predicted temperature, using the recursive least squares method.

[0105] It is easy to understand that, in some embodiments of this application, the active thermal management system of the edge AI chip includes: a load prediction engine, used to acquire the number of input tokens, the number of historical output tokens, the queue depth, and the current inference stage identifier for the inference task, and predict the power consumption trajectory and peak temperature prediction value within multiple future time windows based on a hardware-implemented lightweight recurrent neural network (i.e., the target load prediction neural network model); a thermal trajectory inferrer, used to map the power consumption trajectory to a temperature change trajectory based on the chip thermal model, and generate a pre-regulation trigger signal according to the prediction result; an active collaborative regulation actuator, used to perform at least one of voltage pre-regulation, differentiated frequency regulation, fan pre-acceleration, and accuracy pre-degradation within a preset time window before a load surge; and an online correction loop, used to monitor the deviation between the actual temperature and the predicted temperature, and update the load prediction model and thermal model parameters online. In a preferred embodiment, the edge-side large model is a lightweight model that can be deployed on the edge, such as Gemma or Qwen, and its inference stage includes at least a Prefill stage and a Decode stage. In a preferred embodiment, the load prediction engine is implemented using a gated recurrent unit (GRU), and its weights are quantized using INT4 to reduce hardware area and power consumption. In a preferred embodiment, the thermal trajectory extrapolator employs a first-order RC equivalent thermal model and identifies thermal resistance and thermal capacity parameters online using recursive least squares. In a preferred embodiment, the active collaborative control actuator includes a voltage pre-regulator, a non-uniform frequency regulator, a fan pre-acceleration driver, and a precision pre-degradation buffer. In a preferred embodiment, the online correction loop performs parameter updates during inference task idle periods to avoid interfering with the real-time inference process.

[0106] For example, in some embodiments of this application, the load prediction engine receives information such as the number of input tokens, the number of historical output tokens, queue depth, and stage identifiers. It then uses a hardware-implemented GRU network to predict power consumption trends over multiple future time windows and outputs power consumption trajectories, predicted peak temperatures, and prediction confidence levels. In one implementation, the GRU network uses INT4 quantization and implements activation functions through lookup tables to reduce computational complexity. The thermal trajectory extrapolator performs temperature extrapolation on the power consumption trajectory based on a first-order RC equivalent thermal model and generates a pre-regulation signal based on the relationship between the predicted temperature and a safety threshold. When the predicted peak temperature approaches the threshold, the system dynamically adjusts the safety margin based on the prediction confidence level to improve the accuracy and robustness of the pre-regulation. The active collaborative regulation actuator performs at least one of the following operations based on the pre-regulation signal: increasing the chip supply voltage before a load surge to reduce the risk of power network voltage drop; implementing differentiated frequency regulation for compute unit clusters and memory access unit clusters to alleviate localized thermal loads; increasing the cooling fan speed in advance when the thermal trajectory rise rate exceeds a threshold; and switching to low-precision weights in emergency situations to shorten the duration of load peaks. The online correction loop recursively updates the thermal model parameters based on the error between the actual temperature and the predicted temperature, and performs incremental correction on the prediction model (i.e., the target load prediction neural network model) at appropriate times to adapt to parameter drift during long-term chip operation.

[0107] Some embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, characterized in that the program, when executed by a processor, can implement the methods described in the above embodiments.

[0108] like Figure 3 As shown, some embodiments of this application provide an electronic device 300, including a memory 310, a processor 320, and a computer program stored in the memory and executable on the processor, wherein the processor 320 can implement the method described in any of the above embodiments when it reads the program from the memory 310 via a bus 330 and executes the program.

[0109] Processor 320 can process digital signals and may include various computing architectures. For example, it may be a complex instruction set computer architecture, a reduced instruction set computer architecture, or an architecture that implements multiple instruction set combinations. In some examples, processor 320 may be a microprocessor.

[0110] The memory 310 can be used to store instructions executed by the processor 320 or data related to the execution of instructions. These instructions and / or data may include code used to implement some or all of the functions of one or more modules described in the embodiments of this application. The processor 320 of the embodiments of this disclosure can be used to execute the instructions in the memory 310 to implement… Figure 2 The method shown. Memory 310 includes dynamic random access memory, static random access memory, flash memory, optical memory, or other memory well known to those skilled in the art.

[0111] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0112] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0113] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0114] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application. It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0115] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0116] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. An active thermal management method for edge AI chips, characterized in that, The active thermal management method includes: Obtain the characteristic parameters of the current inference task running in the NPU of the chip to be regulated, wherein the characteristic parameters include: the number of input tokens, the number of historical output tokens, the queue depth of the current inference task, and the inference stage identifier; At least based on the aforementioned feature parameters, a task feature vector is constructed, wherein the task feature vector includes: equivalent computational operands, equivalent memory access data volume, parallelism pressure factor, and the inference stage identifier. The inference stage identifier is an enumerated variable used to indicate the stage of the current inference task. The equivalent computational operands are used to characterize the busyness of the target computing resources of the NPU. The equivalent memory access data volume is an estimate of the total amount of data that the NPU needs to read and write from off-chip memory or on-chip cache when executing the current inference task. The parallelism pressure factor is a dimensionless normalized index used to characterize the amplification effect of the congestion of the current task queue of the NPU on power consumption. The task feature vector is input into the target load prediction neural network model to determine the power consumption trajectory of the NPU in multiple future time windows; The power consumption trajectory is mapped to the temperature change trajectory of the chip to be controlled based on the equivalent thermal model, and a control strategy for adjusting the chip to be controlled is generated based on the temperature change trajectory.

2. The active thermal management method as described in claim 1, characterized in that, The construction of the task feature vector based at least on the feature parameters includes: Obtain the fixed architecture parameters of the edge-side large model running on the NPU, wherein the fixed architecture parameters include: number of layers, hidden dimension, and feedforward network intermediate dimension; If it is confirmed that the inference stage identifier indicates that the end-side large model is in the Prefill stage, then the equivalent computational operands are determined using the following formula: Oh comp = N in × ( 2 × L × (4H² + 2H × D ff) ) × 10⁻ 9 Where, N in Let L be the number of input tokens, H be the number of layers in the large edge model, and D be the hidden dimension of the large edge model. ff 4H² represents the intermediate dimension of the feedforward network in the large end-side model, and 4H² represents the computational cost of Q, K, and V projections and the output projection in each attention mechanism layer; 2H × D ff The computational cost of two fully connected layers in each feedforward network; If it is confirmed that the inference stage identifier indicates that the large end-side model is in the Decode stage, then the equivalent computational operands are determined using the following formula: Oh comp = N outhist × [ 2 × L × (4H² + 2H × D ff ) + 2 × L × H × N in ] × 10⁻ 9 Where, N outhist N represents the number of output tokens generated during the autoregressive decoding process. in Let L be the number of input tokens, H be the number of layers in the large edge model, and D be the hidden dimension of the large edge model. ff is the intermediate dimension of the feedforward network of the large end-side model.

3. The active thermal management method as described in claim 2, characterized in that, The construction of the task feature vector based at least on the feature parameters includes: Mdata = (N in +N outhist ) × L × H × 2 + (Total model weights) × Activation factor Wherein, Mdata is the equivalent memory access data volume, the activation factor takes a first value in the Prefill stage and a second value in the Decode stage, and the second value is less than the first value.

4. The active thermal management method as described in claim 3, characterized in that, The construction of the task feature vector based at least on the feature parameters includes: P = min(Q / Qmax, 1.0) Where Q is the queue depth of the current inference task, Qmax is the preset maximum queue depth, and P is the parallelism pressure factor.

5. The active thermal management method as described in claim 4, characterized in that, The target load prediction neural network model includes an input layer, a GRU hidden layer, and a fully connected output layer. The input layer is used to receive the equivalent computation operands, the equivalent memory access data volume, the parallelism pressure factor, and the inference stage identifier. The GRU hidden layer contains sixteen neurons and uses a gating mechanism to fuse the input with the historical hidden state and capture the temporal dependence of the load dynamics. The output layer includes multiple neurons, and each neuron is used to output the predicted power consumption value at a future time.

6. The active thermal management method as described in claim 5, characterized in that, The active thermal management method further includes: The inference task of the large-scale edge model is started on the chip to be controlled, and various inference loads are constructed. The various inference loads cover various running scenarios from the Prefill stage to the Decode stage by adjusting the values ​​of the controllable parameters of the inference task. The types of controllable parameters include: input prompt length, output generation length and number of concurrent tasks. Before each inference task is issued, the number of input tokens, the number of historical output tokens, the queue depth, and the inference stage identifier corresponding to the current inference task are read from the task scheduler to obtain a four-dimensional feature vector of the sample, which is used as the input feature of a sample and the timestamp is recorded. Starting from the recorded timestamp, the average power consumption of the NPU is continuously collected at preset intervals using a power consumption sensor over multiple subsequent time windows, and the average power consumption value is used as the label value for this sample. The sample's four-dimensional feature vector and the labeled value are bound together to form a training sample and stored in the dataset; The target load prediction neural network model is obtained by training the load prediction neural network model based on the dataset.

7. The active thermal management method as described in claim 6, characterized in that, The step of training the load prediction neural network model based on the dataset to obtain the target load prediction neural network model includes: A batch of samples is randomly selected from the dataset; The four-dimensional feature vectors of the batch of samples are input into the input layer to pass the four-dimensional feature vectors corresponding to the batch of samples to the GRU hidden layer; the GRU hidden layer calculates the gate value and updates the hidden state based on the current weight and the hidden state at the previous time step; the output layer generates multiple predicted power consumption values ​​and a confidence value. The total loss is determined based on the predicted power consumption value, the labeled value, and the loss function, wherein the loss function is composed of the sum of the main loss and the auxiliary loss. The main loss is the mean square error between multiple predicted power consumption values ​​and multiple labeled values ​​of actual power consumption, and the auxiliary loss is the confidence calibration loss. The training parameters are adjusted based on the total loss until the training is completed and the target load prediction neural network model is obtained.

8. The active thermal management method as described in claim 1, characterized in that, The step of mapping the power consumption trajectory to the temperature change trajectory of the chip to be regulated based on the equivalent thermal model, and generating a regulation strategy for adjusting the chip to be regulated based on the temperature change trajectory, includes: Using the current junction temperature of the chip to be controlled as the initial value, the predicted power consumption values ​​of each time window in the power consumption trajectory are sequentially substituted into the equivalent thermal model, and the predicted temperature values ​​of the chip to be controlled in each time window are recursively calculated. The temperature change trajectory is obtained by connecting the predicted temperature values. The temperature risk level is determined based on the peak temperature and temperature rise rate in the temperature change trajectory. At least based on the temperature risk level, generate temperature adjustment strategies corresponding to different inference stages; The model parameters of the equivalent thermal model include equivalent thermal resistance and equivalent heat capacity. The equivalent thermal resistance and equivalent heat capacity are updated online using the recursive least squares method based on the deviation between the actual temperature and the predicted temperature through an online correction loop.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it can implement the active thermal management method as described in any one of claims 1-8.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein, When the processor executes the computer program, it can implement the active thermal management method as described in any one of claims 1-8.

Images