An adaptive load resource scheduling method and device for embedded GPU

Abstract

This invention belongs to the technical field of embedded artificial intelligence and real-time system scheduling, and more specifically, relates to an adaptive load resource scheduling method and apparatus for embedded GPUs. The method involves offline acquisition and modeling of the platform's thermal throttling behavior, constructing DNN variants with different computational loads and accuracies, and profiling them under multiple frequency and TPC configurations to obtain a latency-power mapping table. In the online phase, based on task deadline margins and current temperature / frequency states, the method looks up the variant with the highest accuracy and meeting the deadline under the current thermal conditions and selects the corresponding TPC configuration from the profiling table. If necessary, phased online pruning and low-overhead switching are implemented, and the method collaborates with system-level DVFS / task mapping to optimize energy efficiency. This method can still guarantee the real-time performance of inference tasks under passive frequency reduction conditions and significantly reduce performance fluctuations caused by thermal throttling.

Description

Technical Field

[0001] This invention belongs to the technical field of embedded artificial intelligence and real-time system scheduling, and more specifically, relates to an adaptive load resource scheduling method and apparatus for embedded GPUs. Background Technology

[0002] With the increasing demand for low-latency inference in edge AI applications (such as video stream object detection and autonomous driving perception), embedded SoCs (such as the NVIDIA Jetson series) have become important deployment platforms. Embedded GPUs integrate temperature sensors and hardware thermal managers on-chip. When the node temperature exceeds a threshold, the hardware triggers thermal throttling, forcibly reducing the GPU frequency, leading to a significant increase in inference latency and potentially violating real-time task deadlines. While existing model compression and dynamic network technologies can generate model variants with different workloads, most focus on average latency or throughput optimization and do not combine model-level adaptation with system-level resource allocation (such as TPC activation) under passive thermal throttling for real-time guarantees. System-level DVFS or task mapping methods typically assume that the frequency can be directly controlled by software and cannot directly handle frequency reduction passively triggered by hardware.

[0003] For example, Chinese patent document CN121166377A discloses an edge AI inference computing power-power collaborative system. This system constructs a multi-scheme matrix through task decomposition and computing power / energy consumption prediction, and introduces a mechanism for triggering dynamic task migration by temperature monitoring.

[0004] Chinese patent document CN120354954A discloses a dynamic model switching framework for AI inference optimization methods and systems for edge devices. The framework includes training a quantization compensation model through federated learning to correct quantization errors; constructing a hierarchical heterogeneous resource management system to achieve multi-core collaborative scheduling; integrating compensation parameters into the FPGA acceleration circuit and adopting intelligent power balancing technology; and using a graph neural network to construct a dynamic decision engine to trigger model switching, providing a feasible path for dynamic adjustment at the model level.

[0005] This invention aims to overcome the above-mentioned shortcomings by combining offline profiling with online conservative decision-making to achieve predictable and robust scheduling and energy efficiency management in passive frequency reduction scenarios. Summary of the Invention

[0006] This invention aims to overcome at least one of the defects of the prior art and provide an adaptive load resource scheduling method for embedded GPUs. Under the condition of passive frequency reduction of embedded GPUs, the method can dynamically adjust the DNN load and work in coordination with system-level resources, thereby optimizing energy efficiency and reducing the probability of thermal throttling while ensuring real-time performance. This solves the problems of inference task delay and energy efficiency degradation caused by thermal throttling (passive frequency reduction) of embedded GPUs.

[0007] The present invention also discloses an apparatus loaded with an adaptive load resource scheduling method for embedded GPUs.

[0008] The detailed technical solution of this invention is as follows:

[0009] An adaptive load resource scheduling method for embedded GPUs, applied to an embedded GPU platform with passive thermal throttling characteristics, the method includes an offline preprocessing stage and an online real-time scheduling stage:

[0010] The offline preprocessing stage includes the following steps:

[0011] S1. Collect and analyze the dynamic temperature and frequency changes of the target edge SoC platform under the passive thermal throttling mechanism, establish a semi-empirical model of temperature-frequency-execution time-energy consumption, and obtain the worst execution time boundary of the inference task under different thermal states.

[0012] S2. Select a shared baseline CNN model and construct a set of DNN variants with different computational loads and inference accuracies. Use a shared weight and alternating activation strategy to complete the joint training of all DNN variants and test to obtain the reference latency and accuracy of each DNN variant.

[0013] S3. Perform performance profiling on each variant under multiple GPU frequency levels and TPC activation configurations, and establish a latency table C and a power consumption table P for each variant under different resource configurations. The two together form a profiling table.

[0014] The online real-time scheduling phase includes the following steps:

[0015] S4. When each inference task arrives, the task cutoff margin is calculated, and the GPU frequency during the task execution phase is conservatively evaluated to determine the evaluation frequency level corresponding to the task.

[0016] S5. Based on the task cutoff margin and the frequency level obtained from the evaluation, query the delay table C, select the DNN variant model with the highest accuracy, select the TPC configuration with the fewest activations in the feasible TPC configurations corresponding to the model, and start inference.

[0017] According to a preferred embodiment of the present invention, step S1 is specifically as follows:

[0018] Consult the target edge SoC platform's datasheet and thermal management documentation to clarify the operating parameters of the passive thermal throttling mechanism, including the thermal throttling trigger temperature threshold. A set of frequency ranges corresponding to different temperature ranges Control strategies for frequency drop and recovery, and the types of temperature and frequency sensor interfaces that the system can read;

[0019] Design a single-variable control experiment. While maintaining consistency in AI inference task type, model structure, and operating environment, gradually change the single adjustment of load intensity or heat dissipation conditions, and periodically collect data tuples { , , The corresponding GPU inference time and total platform power consumption are used to form a sample set corresponding to the GPU frequency state and task inference time. as well as The three tuple variables are the current motherboard temperature, GPU temperature, and GPU clock frequency, respectively.

[0020] Based on the sample set data, a semi-empirical model of temperature-frequency-execution time-energy consumption is established to obtain the distribution characteristics of task inference execution time, obtain the frequency stability segment, frequency reduction trigger point and recovery lag characteristics under each temperature zone, and extract its worst execution time.

[0021] According to a preferred embodiment of the present invention, step S2 is specifically as follows:

[0022] Select a shared baseline CNN model and construct a set of DNN variants M={ ,…, Variant models share some parameters with the baseline model;

[0023] A shared weight and alternating activation strategy is used to train the DNN variant: first, a complete shared baseline CNN model is trained to obtain shared initial weights; then, in one training cycle, a certain parameter ratio is randomly activated with a set probability. The corresponding subnetworks are propagated forward and backward, and the shared parameters are updated synchronously among all activated subnetworks, while the specific parameters are updated only when the corresponding subnetwork is activated; then the model parameters of all DNN variants are fine-tuned, taking into account the accuracy of the output of all variants in the fine-tuning process.

[0024] After training, inference tests are performed on each variant on the target edge SoC platform, and the reference latency and accuracy of each variant are recorded.

[0025] According to a preferred embodiment of the present invention, step S3 is specifically as follows:

[0026] For each variant Selected frequency ranges offline With several TPC mask configurations ∈ Run representative inputs and record the maximum observed GPU inference latency. and corresponding power consumption Where k is the DNN variant index, j is the frequency range index, and h is the TPC configuration index;

[0027] All data is organized into a delay table C and a power consumption table P, which together form a profiling table. Variants are then reordered and dominant items are removed based on reference delay and accuracy.

[0028] According to a preferred embodiment of the present invention, step S4 is specifically as follows:

[0029] When each inference task arrives or a job begins, read the current time t and the job start time. Absolute deadline Calculate the task deadline margin Simultaneously, read the current GPU report frequency level index j and node temperature T(t), and preset the temperature margin. ,like If the frequency reduction is about to be triggered, then set the evaluation frequency j' to a lower level than the current frequency j in the Profiling table; otherwise, set j' = j.

[0030] According to a preferred embodiment of the present invention, step S5 is specifically as follows:

[0031] Given j', search for the condition that satisfies the delay table C. ≤ The feasible triples are used to obtain a set of feasible DNN variant models;

[0032] Select the DNN variant model with the highest accuracy from the set of feasible DNN variant models. Then, select the TPC configuration with the fewest activations from the feasible TPC set corresponding to the model. If the set of feasible models is empty, then mark the job as unschedulable.

[0033] Select to ( , After that, the model index is applied atomically at the task boundary. With TPC configuration Initiate reasoning.

[0034] According to a preferred embodiment of the present invention, step S5 further includes dynamic adjustment during the inference process: if progress lag or rapid temperature rise is detected during the inference process, the scheduler, starting from the remaining network stages, reduces the load α of the remaining stages according to predetermined rules to ensure that the remaining execution time is satisfied. ≤ .

[0035] According to a preferred embodiment of the present invention, the online real-time scheduling phase further includes system coordination and adaptive adjustment, specifically as follows:

[0036] The online scheduler works in conjunction with the system-level DVFS and task mapping module: when the system requires reduced power consumption or the temperature reaches a critical point, the scheduler synchronously reduces the load α and allows the DVFS to moderately reduce its frequency; when resource distribution allows, parallel subtasks are mapped to computing clusters that are more energy efficient at the current temperature to achieve pipelined parallelism.

[0037] The system continuously records historical statistics. If the GPU frequency is passively reduced within the set time window, the scheduling strategy will prioritize a more conservative j' or a lower load α to ensure long-term stability and energy efficiency.

[0038] According to a preferred embodiment of the present invention, the scheduler during the operation of the method only needs to traverse and perform table lookups on K models and H TPC configurations in the worst case, with a complexity of O(K + H); all variants reside in GPU memory, and variant model switching only requires updating the index or activation mask, and is applied atomically at the task boundary; the construction of the profiling table can be parallelized by sampling, and the selection of table entries should adopt the conservative WCET.

[0039] In another aspect of the invention, an apparatus is provided that is loaded with an adaptive load resource scheduling method for embedded GPUs, the apparatus comprising an offline preprocessing module and an online real-time scheduling module;

[0040] The offline preprocessing module includes a passive down-frequency analysis unit, a DNN variant construction unit, and a profiling table building unit;

[0041] The passive frequency reduction analysis unit is used to collect and analyze the dynamic temperature and frequency change characteristics of the target edge SoC platform under the passive thermal throttling mechanism, establish a semi-empirical model of temperature-frequency-execution time-energy consumption, and obtain the worst execution time boundary of the inference task under different thermal states.

[0042] The DNN variant building unit is used to select a shared baseline CNN model and build a set of DNN variants. It uses a shared weight and alternating activation strategy to complete the joint training of all DNN variants and tests to obtain the reference latency and accuracy of each DNN variant.

[0043] The Profiling table building unit is used to perform performance profiling on each variant under multiple GPU frequency levels and TPC activation configurations, and to build a latency table C and a power consumption table P containing each variant under different resource configurations.

[0044] The online real-time scheduling module includes a runtime monitoring and evaluation unit, a scheduling decision unit, and a system collaborative adaptive unit;

[0045] The runtime monitoring and evaluation unit calculates the task cutoff margin when each inference task arrives, and at the same time performs a conservative evaluation of the GPU frequency during the task execution phase to determine the evaluation frequency level corresponding to the task.

[0046] The scheduling decision unit, based on the task deadline margin and the frequency level obtained from the evaluation, queries the delay table C, selects the DNN variant model with the highest accuracy, selects the TPC configuration with the fewest activations from the feasible TPC configurations corresponding to the model, and starts inference.

[0047] The system collaborative adaptive unit is used to collaboratively optimize the online scheduling process with the system-level DVFS module and task mapping module, while continuously recording historical statistical information of task operation, and making closed-loop adaptive adjustments to the scheduling strategy based on the historical statistical information.

[0048] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0049] (1) The present invention explicitly describes the mapping relationship of "temperature-frequency-execution time-energy consumption-accuracy", allowing the scheduler to make predictable decisions based on the profiling table under passive frequency reduction conditions.

[0050] (2) This invention combines model-level adaptation (multi-variant / staged pruning) with hardware-level resource configuration (TPC activation / frequency) for real-time scheduling, thereby significantly reducing instantaneous power consumption and reducing the probability of thermal throttling triggering while ensuring the deadline.

[0051] (3) The present invention incorporates historical statistics into the adaptive strategy in a closed loop, so that energy efficiency and stability can be continuously optimized in long-term operation scenarios.

[0052] (4) The runtime decision-making of this invention is only table lookup and linear scan, with low time complexity (O(K+H)), and the robustness to passive frequency reduction is improved by conservative frequency evaluation and task boundary atomic update strategy. Attached Figure Description

[0053] Figure 1 This is a flowchart of the adaptive load resource scheduling method for embedded GPUs described in this invention.

[0054] Figure 2 This is a schematic diagram of the overall structure of the load resource scheduling method described in Embodiment 1 of the present invention.

[0055] Figure 3 This is a schematic diagram of the embedded GPU architecture and task model described in Embodiment 1 of the present invention. Detailed Implementation

[0056] The present disclosure will be further described below with reference to the accompanying drawings and embodiments.

[0057] This invention provides an adaptive load resource scheduling method for embedded GPUs, applicable to embedded GPU platforms with passive thermal throttling characteristics (such as NVIDIA Jetson AGX Xavier, Jetson Orin, etc.). The overall process of this invention relies on... Figure 3 The embedded GPU architecture and task model implementation shown; Figure 3 Part (a) shows the embedded GPU (Jetson-type) module structure and TPC / SM organization. Figure 3 Part (b) illustrates the scenario where rapid temperature rise is caused by short-term execution and high instantaneous power under full TPC activation. Figure 3 Part (c) shows the case where inference is prolonged but instantaneous power and temperature rise are reduced under partial TPC activation;

[0058] On embedded GPU platforms such as the NVIDIA Jetson AGX Xavier, the CPU and GPU are integrated on the same SoC and communicate via a high-bandwidth on-chip interconnect. Figure 3 As shown in section (a), the GPU consists of multiple TPCs, each containing two Streaming Multiprocessors (SMs). DNN operators such as convolution and matrix multiplication can be executed in parallel on these SMs, providing high throughput for inference workloads. The SoC integrates an on-chip temperature sensor and a hardware thermal management controller. When the temperature exceeds a threshold, the controller triggers thermal throttling by reducing the allowed GPU frequency range and forcing the GPU to run at a lower clock frequency; in extreme cases, the frequency may be further reduced, even triggering a protective reset. For DNN inference, this manifests as a passive DVFS: the DNN workload itself remains unchanged, but the GPU frequency is hardware-reduced, leading to increased DNN inference latency and potentially violating real-time deadline requirements.

[0059] Figure 3 Sections (b) and (c) illustrate the motivation behind the scheduling strategy of this invention: Based on the analysis of the passive frequency reduction mechanism of the GPU caused by thermal throttling and its consequences, the motivation of this invention stems from the following key motivations:

[0060] 1. DNN Workload Adaptation: The accumulation of heat within the system is determined by the balance between power dissipation and cooling efficiency. Intuitively, to counter thermal throttling, inference must still be completed despite performance degradation, so heat generation must be reduced by lowering the computational load. Modern DNNs can be designed to explicitly exhibit a "latency-accuracy" trade-off. The scheduler can select a lighter variant at runtime based on the current frequency and slack time such that its inference latency is less than the GPU execution time . As long as the condition ( ≤ ) is always satisfied, this strategy can reduce the GPU execution time and resume meeting the deadline after a heat-induced frequency drop.

[0061] 2. Power Shaping Based on TPC Activation: At the system level, transient heat generation can be reduced by executing tasks more slowly. As shown in part (b) of Figure 3 , when all TPCs are activated, each layer of the DNN can be more aggressively parallelized across more SMs, minimizing the GPU execution time. However, this configuration also results in the highest transient power consumption, faster temperature rise, and an increased probability of triggering thermal throttling. Conversely, as shown in part (c) of Figure 3 , when the task has sufficient slack time, the scheduler can deliberately stretch its execution by reducing the number of active TPCs: at the cost of a moderate increase in the GPU execution time, in exchange for a significant reduction in transient power consumption and a lower probability of future thermal throttling.

[0062] The following further describes an adaptive load resource scheduling method and apparatus for an embedded GPU according to the present invention in combination with specific embodiments.

[0063] Embodiment 1

[0064] Refer to Figure 1 and Figure 2 This embodiment provides an adaptive load resource scheduling method for an embedded GPU. The method includes an offline preprocessing stage and an online real-time scheduling stage, achieving real-time guarantee for DNN inference tasks and system energy efficiency optimization in passive frequency drop scenarios, while reducing the probability of thermal throttling trigger, specifically as follows:

[0065] Offline Preprocessing Stage (Steps S1–S3)

[0066] S1. Passive Frequency Reduction Mechanism Analysis and Data Acquisition: Collect and analyze the dynamic temperature and frequency changes of the target edge SoC platform under the passive thermal throttling mechanism, establish a semi-empirical model of temperature-frequency-execution time-energy consumption, and obtain the worst-case execution time boundary of the inference task under different thermal states, as detailed below:

[0067] Consult the target edge SoC platform's datasheet and thermal management documentation to clarify the operating parameters of the passive thermal throttling mechanism, including the thermal throttling trigger temperature threshold. A set of frequency ranges corresponding to different temperature ranges The steps outline the control strategies for frequency reduction and recovery, as well as the types of temperature and frequency sensor interfaces that the system can read. These steps are used to confirm whether the GPU possesses passive thermal regulation capabilities based on its internal power management module. Specifically, when the GPU temperature continues to rise, the system will dynamically reduce the GPU's operating frequency, and when the temperature drops, the GPU frequency will automatically increase within acceptable limits. This frequency regulation process is autonomously completed by the internal chip module, presenting a dynamic frequency change behavior that cannot be directly controlled by the operating system and upper-layer applications.

[0068] Based on this, a single-variable control experiment was designed. While maintaining consistency in the AI ​​inference task type, model structure, and operating environment, the load intensity or heat dissipation conditions (such as fan speed or input frame rate) were gradually changed to keep the GPU temperature within different thermal ranges over a longer period. This caused the GPU to experience a dynamic frequency reduction and recovery process during operation. Data tuples were periodically collected during this experiment. , , The corresponding GPU inference time and total platform power consumption are used to form a sample set corresponding to the GPU frequency state and task inference time. as well as The three tuple variables are the current motherboard temperature, GPU temperature, and GPU clock frequency.

[0069] Based on a large sample set of data, a semi-empirical model of temperature-frequency-execution time-energy consumption was fitted and established to obtain the distribution characteristics of task inference execution time, obtain the frequency stability segment, frequency reduction trigger point and recovery lag characteristics under each temperature zone, and extract its worst execution time.

[0070] The above execution time statistics are used to characterize the objective performance degradation of AI tasks under passive GPU downclocking conditions, providing execution time reference boundaries for task scheduling and model switching decisions based on thermal state awareness in subsequent steps, thereby avoiding AI inference tasks exceeding system timing constraints due to dynamic changes in GPU frequency.

[0071] S2. DNN Variants Construction and Training: A shared baseline CNN model was selected, and a set of DNN variants were constructed. All DNN variants were jointly trained using a shared weight and alternating activation strategy. The reference latency and accuracy of each DNN variant were tested, as detailed below:

[0072] Select a shared baseline CNN model (e.g., ResNet-101) and construct a set of DNN variants M={ ,…, Variant models can be obtained through methods such as structured channel pruning, residual block skipping, multiple exits / early exits, lightweight replacement layers, or wide-depth scaling. Variant models share some parameters with the baseline model.

[0073] A shared weight and alternating activation strategy is used to train the DNN variant: first, a complete shared baseline CNN model is trained to obtain shared initial weights; then, in one training cycle, a certain parameter ratio is randomly activated with a set probability. The corresponding subnetworks undergo forward and backward propagation, with shared parameters updated synchronously across all activated subnetworks, while specific parameters are updated only when the corresponding subnetwork is activated. Based on the high accuracy of the baseline CNN model, the model parameters of all DNN variants are then fine-tuned. The fine-tuning process takes into account the accuracy of the outputs of all variants, ensuring that each variant has the optimal accuracy under its own computational load.

[0074] After training, inference tests are performed on each variant on the target edge SoC platform, and the reference latency and accuracy of each variant are recorded.

[0075] S3. Profiling and Table Construction: Performance profiling is performed on each variant under multiple GPU frequency levels and TPC activation configurations. A latency table C and a power consumption table P containing each variant under different resource configurations are built. The two tables together form the profiling table, as detailed below:

[0076] For each variant Selected frequency ranges offline With several TPC mask configurations ∈ Run representative inputs and record the maximum observation GPU inference latency. and corresponding power consumption Where k is the DNN variant index, j is the frequency range index, and h is the TPC configuration index;

[0077] All data is organized into a latency table C and a power consumption table P, which together form a profiling table. Variants are then reordered and dominant terms are removed based on reference latency and accuracy to ensure the monotonicity of the sorted DNN variant list (the larger the index, the higher the latency and accuracy).

[0078] Online real-time scheduling phase (steps S4–S6)

[0079] S4. Runtime Monitoring and Conservative Frequency Assessment: When each inference task arrives, the task cutoff margin is calculated, and the GPU frequency during the task execution phase is conservatively assessed to determine the corresponding assessment frequency level for the task, as detailed below:

[0080] When each inference task arrives or a job begins, read the current time t and the job start time. Absolute deadline Calculate the task deadline margin .

[0081] Simultaneously read the current GPU report frequency level index j and node temperature T(t), with a preset temperature margin ΔT. If T(t) ≥ If ΔT indicates that frequency reduction is about to be triggered, then set the evaluation frequency j' to a lower level than the current frequency j in the Profiling table (such as the next most recent level); otherwise, set j' = j.

[0082] S5. DNN Variant Model and TPC Selection: Based on the task cutoff margin and the frequency range obtained from the evaluation, query the delay table C, select the DNN variant model with the highest accuracy, and select the TPC configuration with the fewest activations from the feasible TPC configurations corresponding to this model, and start inference, as follows:

[0083] Given j', search for the condition that satisfies the delay table C. ≤ The feasible triples are used to obtain a set of feasible DNN variant models.

[0084] Scheduling targets are ordered lexicographically: First, select the DNN variant model with the highest accuracy from the set of feasible DNN variant models (equivalent to selecting the DNN variant with the largest index). Then, select the TPC configuration with the fewest activations from the set of feasible TPCs corresponding to the model. If the set of feasible models is empty, the job is marked as unschedulable, for example, by triggering a degradation policy, task dropping, or an alarm.

[0085] Select to ( , After that, the model index is applied atomically at the task boundary. With TPC configuration Inference is then initiated. If progress lag or a rapid temperature rise is detected during inference, the scheduler can start from the remaining network phases and reduce the load α of the remaining phases according to predetermined rules (e.g., reducing the number of reserved channels or selecting shallow exits in advance) to ensure that the remaining execution time meets the requirements. ≤ .

[0086] The online real-time scheduling phase also includes system coordination and adaptive adjustment, as detailed below:

[0087] The online scheduler works in conjunction with the system-level DVFS and task mapping module: when the system requires reduced power consumption or the temperature reaches a critical point, the scheduler synchronously reduces the load α and allows the DVFS to moderately reduce its frequency; when resource distribution allows, parallel subtasks are mapped to computing clusters that are more energy efficient at the current temperature to achieve pipelined parallelism.

[0088] The system continuously records historical statistics (such as actual execution time, number of frequency reductions, and temperature rise rate). If the GPU frequency continues to be passively reduced within the set time window, the scheduling strategy will prioritize a more conservative j' or a lower load α to ensure long-term stability and energy efficiency.

[0089] Algorithm complexity and implementation details:

[0090] Each decision by the runtime scheduler only requires traversing and performing table lookups on K models and H TPC configurations in the worst case, with a complexity of O(K + H), where O represents the algorithm's time complexity. All variants reside in GPU memory (or a memory pool), and variant model switching only requires updating the index or activation mask, resulting in extremely low switching overhead and atomic application at task boundaries to avoid interference during flight. The construction of the profiling table can be parallelized, and table entry selection should adopt a conservative WCET (maximum observation delay or with a safety margin) to ensure online schedulability. For the implementation layer, it is necessary to support CUDA / inference operators that skip layers / channels by mask during inference runtime (or use lightweight alternative operators), and ensure high reliability on embedded platforms.

[0091] Example 2

[0092] This embodiment provides an apparatus loaded with an adaptive load resource scheduling method for embedded GPUs, the apparatus including an offline preprocessing module and an online real-time scheduling module;

[0093] The offline preprocessing module includes a passive down-frequency analysis unit, a DNN variant construction unit, and a profiling table building unit.

[0094] The passive frequency reduction analysis unit is used to collect and analyze the dynamic temperature and frequency change characteristics of the target edge SoC platform under the passive thermal throttling mechanism, establish a semi-empirical model of temperature-frequency-execution time-energy consumption, and obtain the worst execution time boundary of the inference task under different thermal states.

[0095] The DNN variant building unit is used to select a shared baseline CNN model and build a set of DNN variants. It uses a shared weight and alternating activation strategy to complete the joint training of all DNN variants and tests to obtain the reference latency and accuracy of each DNN variant.

[0096] The Profiling table building unit is used to perform performance profiling on each variant under multiple GPU frequency levels and TPC activation configurations, and to build a latency table C and a power consumption table P containing each variant under different resource configurations.

[0097] The online real-time scheduling module includes a runtime monitoring and evaluation unit, a scheduling decision unit, and a system collaborative adaptive unit;

[0098] The runtime monitoring and evaluation unit calculates the task cutoff margin when each inference task arrives, and at the same time performs a conservative evaluation of the GPU frequency during the task execution phase to determine the evaluation frequency level corresponding to the task.

[0099] The scheduling decision unit, based on the task deadline margin and the frequency level obtained from the evaluation, queries the delay table C, selects the DNN variant model with the highest accuracy, selects the TPC configuration with the fewest activations from the feasible TPC configurations corresponding to the model, and starts inference.

[0100] The system collaborative adaptive unit is used to collaboratively optimize the online scheduling process with the system-level DVFS module and task mapping module, while continuously recording historical statistical information of task operation, and making closed-loop adaptive adjustments to the scheduling strategy based on the historical statistical information.

[0101] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the technical solutions of the present invention, and are not intended to limit the specific implementation of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the claims of the present invention should be included within the protection scope of the claims of the present invention.

Claims

1. An adaptive load resource scheduling method for embedded GPUs, applied to embedded GPU platforms with passive thermal throttling characteristics, characterized in that, The method includes an offline preprocessing stage and an online real-time scheduling stage: The offline preprocessing stage includes the following steps: S1. Collect and analyze the dynamic temperature and frequency changes of the target edge SoC platform under the passive thermal throttling mechanism, establish a semi-empirical model of temperature-frequency-execution time-energy consumption, and obtain the worst execution time boundary of the inference task under different thermal states. S2. Select a shared baseline CNN model and construct a set of DNN variants with different computational loads and inference accuracies. Use a shared weight and alternating activation strategy to complete the joint training of all DNN variants and test the reference latency and accuracy of each DNN variant. S3. Perform performance profiling on each variant under multiple GPU frequency levels and TPC activation configurations, and establish a latency table C and a power consumption table P for each variant under different resource configurations. The two together form a profiling table. The online real-time scheduling phase includes the following steps: S4. When each inference task arrives, the task cutoff margin is calculated, and the GPU frequency during the task execution phase is conservatively evaluated to determine the evaluation frequency level corresponding to the task. S5. Based on the task cutoff margin and the frequency level obtained from the evaluation, query the delay table C, select the DNN variant model with the highest accuracy, select the TPC configuration with the fewest activations in the feasible TPC configurations corresponding to the model, and start inference.

2. The adaptive load resource scheduling method for embedded GPUs according to claim 1, characterized in that, Step S1 is as follows: Consult the target edge SoC platform's datasheet and thermal management documentation to clarify the operating parameters of the passive thermal throttling mechanism, including the thermal throttling trigger temperature threshold. A set of frequency ranges corresponding to different temperature ranges Control strategies for frequency drop and recovery, and the types of temperature and frequency sensor interfaces that the system can read; Design a single-variable control experiment, and while maintaining consistency in the AI ​​inference task type, model structure, and operating environment, gradually change the single variable and periodically collect data tuples { , , The corresponding GPU inference time and total platform power consumption are used to form a sample set corresponding to the GPU frequency state and task inference time. as well as The three tuple variables are the current motherboard temperature, GPU temperature, and GPU clock frequency, respectively. Based on the sample set data, a semi-empirical model of temperature-frequency-execution time-energy consumption is established to obtain the distribution characteristics of task inference execution time, obtain the frequency stability segment, frequency reduction trigger point and recovery lag characteristics under each temperature zone, and extract its worst execution time.

3. The adaptive load resource scheduling method for embedded GPUs according to claim 1, characterized in that, Step S2 is as follows: Select a shared baseline CNN model and construct a set of DNN variants M={ ,…, Variant models share some parameters with the baseline model; A shared weight and alternating activation strategy is used to train the DNN variant: first, a complete shared baseline CNN model is trained to obtain shared initial weights; then, in one training cycle, a certain parameter ratio is randomly activated with a set probability. The corresponding subnetworks are propagated forward and backward, and the shared parameters are updated synchronously among all activated subnetworks, while the specific parameters are updated only when the corresponding subnetwork is activated; then the model parameters of all DNN variants are fine-tuned, taking into account the accuracy of the output of all variants in the fine-tuning process. After training, inference tests are performed on each variant on the target edge SoC platform, and the reference latency and accuracy of each variant are recorded.

4. The adaptive load resource scheduling method for embedded GPUs according to claim 3, characterized in that, Step S3 is as follows: For each variant Selected frequency ranges offline With several TPC mask configurations ∈ Run representative inputs and record the maximum observed GPU inference latency. and corresponding power consumption Where k is the DNN variant index, j is the frequency range index, and h is the TPC configuration index; All data is organized into a delay table C and a power consumption table P, which together form a profiling table. Variants are then reordered and dominant items are removed based on reference delay and accuracy.

5. The adaptive load resource scheduling method for embedded GPUs according to claim 1, characterized in that, Step S4 is as follows: When each inference task arrives or a job begins, read the current time t and the job start time. Absolute deadline Calculate the task deadline margin Simultaneously, read the current GPU report frequency level index j and node temperature T(t), and preset the temperature margin. ,like If the frequency reduction is about to be triggered, then set the evaluation frequency j' to a lower level than the current frequency j in the Profiling table; otherwise, set j' = j.

6. The adaptive load resource scheduling method for embedded GPUs according to claim 5, characterized in that, Step S5 is as follows: Given j', search for the condition that satisfies the delay table C. ≤ The feasible triples are used to obtain a set of feasible DNN variant models; Select the DNN variant model with the highest accuracy from the set of feasible DNN variant models. Then, select the TPC configuration with the fewest activations from the set of feasible TPC configurations corresponding to the model. If the set of feasible models is empty, then mark the job as unschedulable. Select to ( , After that, the model index is applied atomically at the task boundary. With TPC configuration Initiate reasoning.

7. The adaptive load resource scheduling method for embedded GPUs according to claim 6, characterized in that, Step S5 also includes dynamic adjustments during inference: if progress lag or rapid temperature rise is detected during inference, the scheduler, starting from the remaining network stages, reduces the load ratio α of the remaining stages according to predetermined rules to ensure that the remaining execution time meets the requirements. ≤ .

8. The adaptive load resource scheduling method for embedded GPUs according to claim 1, characterized in that, The online real-time scheduling phase also includes system coordination and adaptive adjustment, as detailed below: The online scheduler works in conjunction with the system-level DVFS and task mapping module: when the system requires reduced power consumption or the temperature reaches a critical point, the scheduler synchronously reduces the load α and allows the DVFS to moderately reduce its frequency; when resource distribution allows, parallel subtasks are mapped to computing clusters that are more energy efficient at the current temperature to achieve pipelined parallelism. The system continuously records historical statistics. If the GPU frequency is passively reduced within the set time window, the scheduling strategy will prioritize the more conservative j' or the lower load α.

9. The adaptive load resource scheduling method for embedded GPUs according to claim 1, characterized in that, The scheduler only needs to traverse the K models and H TPC configurations and perform table lookups in the worst case for each decision made by the method, with a complexity of O(K + H). All variants reside in GPU memory, and variant model switching only requires updating the index or activation mask, and is applied atomically at task boundaries. The construction of the profiling table can be parallelized by sampling, and the selection of table entries should adopt the conservative WCET.

10. An apparatus for implementing an adaptive load resource scheduling method for embedded GPUs, characterized in that, The device includes an offline preprocessing module and an online real-time scheduling module; The offline preprocessing module includes a passive down-frequency analysis unit, a DNN variant construction unit, and a profiling table building unit; The passive frequency reduction analysis unit is used to collect and analyze the dynamic temperature and frequency change characteristics of the target edge SoC platform under the passive thermal throttling mechanism, establish a semi-empirical model of temperature-frequency-execution time-energy consumption, and obtain the worst execution time boundary of the inference task under different thermal states. The DNN variant building unit is used to select a shared baseline CNN model and build a set of DNN variants. It uses a shared weight and alternating activation strategy to complete the joint training of all DNN variants and tests to obtain the reference latency and accuracy of each DNN variant. The Profiling table building unit is used to perform performance profiling on each variant under multiple GPU frequency levels and TPC activation configurations, and to build a latency table C and a power consumption table P containing each variant under different resource configurations. The online real-time scheduling module includes a runtime monitoring and evaluation unit, a scheduling decision unit, and a system collaborative adaptive unit; The runtime monitoring and evaluation unit calculates the task cutoff margin when each inference task arrives, and at the same time performs a conservative evaluation of the GPU frequency during the task execution phase to determine the evaluation frequency level corresponding to the task. The scheduling decision unit, based on the task deadline margin and the frequency level obtained from the evaluation, queries the delay table C, selects the DNN variant model with the highest accuracy, selects the TPC configuration with the fewest activations from the feasible TPC configurations corresponding to the model, and starts inference. The system collaborative adaptive unit is used to collaboratively optimize the online scheduling process with the system-level DVFS module and task mapping module, while continuously recording historical statistical information of task operation, and making closed-loop adaptive adjustments to the scheduling strategy based on the historical statistical information.

Images