A business hall scene AI task scheduling method based on computing power grading

Abstract

This invention discloses a scenario-based AI task scheduling method for business halls based on computing power tiers. The method extracts scenario feature vectors, determines computing power demand level labels through computing power consumption calculation and three-level threshold classification, triggers container image pre-fetching and GPU space partitioning configuration; predicts the latency of each task on each path using a heterogeneous execution latency prediction model library, and employs a batch critical path minimization strategy with tiered preference penalties for task assignment; incrementally updates the predictor after filtering outliers from the measured latency; and obtains energy-saving action instructions online through adaptive trade-off parameter solving based on a sliding window-limited dual-deficit queue. This invention solves the problems of computing power utilization imbalance caused by static binding of algorithm carriers and rigidity of energy-saving decisions in preset time slots, achieving a power consumption reduction of over 30% while ensuring real-time SLO.

Description

Technical Field

[0001] This invention relates to the fields of edge computing and artificial intelligence video analysis technology, specifically to a method for dynamic scheduling of computing resources for concurrent scenarios involving multiple cameras and multiple algorithms in business halls. Background Technology

[0002] With the increasing intelligence of service halls in industries such as finance, telecommunications, and government, it has become commonplace for various algorithms based on AI video analytics, including customer flow statistics, facial recognition, behavior recognition, VIP identification, queue monitoring, and abnormal behavior detection, to run concurrently at the edge of these service halls. Due to the vast differences in the size of service halls, with the number of cameras ranging from 1 to 50, the complexity of the algorithms varies greatly. From basic target detection to multimodal complex scene analysis, how to dynamically schedule and control computing resources at edge nodes has become a key technical challenge in this field.

[0003] A common technical solution is to deploy a fixed-configuration edge computing server in each branch office, running a unified AI algorithm stack, with all video streams fed into GPUs for inference. Alternatively, there are cloud-based centralized processing solutions that transmit all camera video back to a central data center for AI analysis.

[0004] The most relevant prior art to this invention is a unified computing power supply solution for edge computing nodes. This solution deploys a fixed-specification GPU server in each business hall, decodes all connected camera video streams at a fixed resolution, and then sends them to the same algorithm container for full algorithm inference. Regarding the binding of the container and the underlying computing power, a pre-defined rule is used to statically map the algorithm type to either CPU or GPU; for example, object detection always runs on GPU, and OCR always runs on CPU. In terms of GPU resource usage, a dedicated GPU or simple MPS time-division multiplexing method is used, without fine-grained slicing of the GPU space. For energy consumption control, a pre-defined daytime high-load and nighttime low-load schedule is used in conjunction with a fixed utilization threshold for energy-saving switching.

[0005] The aforementioned existing technology has the following technical problems:

[0006] Technical Issue 1: Static binding of algorithms and carriers, coupled with coarse-grained use of GPU resources, leads to an imbalance in the utilization of heterogeneous computing power. Service halls vary greatly in size, with the number of cameras ranging from 1 to 50, and algorithm complexity varies widely, from basic recognition to multimodal composite analysis. Existing technologies bind algorithms to CPUs or GPUs based on static mapping rules, and GPUs are either exclusively used for the entire card or simply time-division multiplexed. This fails to dynamically determine the optimal carrier and GPU slice size based on the actual input size, batch size, and concurrency of the current batch of tasks. This frequently results in cross-carrier mismatches where GPUs are fully loaded and queuing while CPUs are idle, and internal GPU slack waste where a single algorithm occupies the entire card but its utilization is less than 50%, worsening inference latency and reducing hardware utilization.

[0007] Technical Issue 2: Reliance on Preset Time Schedules for Energy-Saving Decisions Leads to Rigid Energy Consumption-Real-Time Performance Trade-offs. Existing technologies trigger energy saving using fixed work and rest schedules and utilization thresholds. This fails to adapt when actual load deviates from expected schedules, such as temporary store closures during holidays or promotional activities extending into the night. Furthermore, the lack of modeling for long-term real-time SLO constraints when switching energy-saving actions means that overly aggressive energy saving leads to SLO defaults, forcing a revert to a conservative reserve strategy and resulting in persistently high energy consumption at edge nodes. Summary of the Invention

[0008] The purpose of this invention is to provide a scenario-based AI task scheduling method for business halls based on computing power grading and heterogeneous execution latency prediction, so as to solve the technical problems of imbalance in heterogeneous computing power utilization and rigid energy-saving decision-making in the existing technology.

[0009] This invention provides a scenario-based AI task scheduling method for business halls based on computing power tiers, comprising:

[0010] Based on the video stream access list and algorithm subscription configuration list of the business hall, the scene feature vector is extracted and classified according to the computing power consumption calculation and preset threshold. The computing power demand level label is determined, and the container image pre-pull and graphics processor space partition configuration of the corresponding level are triggered to obtain the pre-ready resource set.

[0011] Based on the pre-ready resource set and the computing power requirement level label, the predicted latency set of each scheduled task on the central processing unit container path and the graphics processing unit slice path is predicted by the heterogeneous execution latency prediction model library. Tasks with insufficient prediction samples are downgraded and assigned according to the computing power requirement level label to obtain a heterogeneous latency prediction result set.

[0012] Based on the heterogeneous latency prediction result set, a batch critical path minimization strategy with hierarchical preference penalty and fragment awareness is adopted to perform task-carrier-slice assignment, and the corresponding container is launched through the container orchestration engine to form a heterogeneous running container set.

[0013] Based on the heterogeneous set of runtime containers, video stream inference is performed, and the measured latency is filtered for outliers and fed back to the heterogeneous execution latency prediction model library to output a standardized business event stream.

[0014] For the standardized business event flow and real-time power consumption, a limited-length real-time deficit queue and a limited-length energy consumption deficit queue are maintained. The energy-saving action command is obtained by online solution with adaptive trade-off parameters and drift penalty.

[0015] According to the energy-saving action instructions, the computing power layer performs container start-up and shutdown and adjusts the graphics processor slicing frequency or sleep mode to form the computing power state after energy saving, which serves as the feedback input for the scene feature vector extraction in the next cycle.

[0016] Further, the step of extracting scene feature vectors based on the business hall video stream access list and algorithm subscription configuration list, and classifying them according to computing power consumption calculation and preset thresholds to determine computing power demand level labels, and triggering container image pre-fetching and graphics processor space partitioning configuration for the corresponding level to obtain a pre-ready resource set, includes: performing structured parsing on the business hall video stream access list and the algorithm subscription configuration list to extract scene feature vectors containing the number of camera channels, resolution of each channel, frame rate of each channel, and set of subscription algorithm types; based on the scene feature vectors, mapping each subscription algorithm type in the set of subscription algorithm types to three complexity levels—basic recognition, multimodal analysis, and high-precision composite scene—through a preset algorithm complexity mapping table, and multiplying and weighting the resolution of each channel, the frame rate of each channel, and the complexity level of the corresponding algorithm to obtain a comprehensive computing power demand value; comparing the comprehensive computing power demand value with a preset first computing power threshold and a second computing power threshold, wherein the first computing power threshold is less than the second computing power threshold, and the comprehensive computing power demand value is less than the first computing power threshold. The following criteria are applied to determine the computing power requirement level: a threshold value indicates a low computing power requirement; a value between the first and second computing power thresholds indicates a medium computing power requirement; and a value greater than the second computing power threshold indicates a high computing power requirement. The computing power requirement level label is then output. Based on the computing power requirement level label, the level-resource mapping table is queried. For the low computing power requirement level, a CPU container image containing only basic recognition-type inference processes is pre-fetched, and the GPU is divided into multiple slice partitions according to the first maximum memory capacity or placed into hibernation. For the medium computing power requirement level, a CPU container image and a GPU slice container image are jointly pre-fetched, and the GPU is divided into mixed-specification slice partitions according to the second maximum memory capacity, where the second maximum memory capacity is greater than the first maximum memory capacity. For the high computing power requirement level, a GPU container image containing complete multimodal analysis and high-precision composite scene inference processes is pre-fetched, and the GPU is divided into slice partitions according to the third maximum memory capacity or configured as a dedicated GPU, where the third maximum memory capacity is greater than the second maximum memory capacity. The pre-ready resource set is then obtained.

[0017] Further, the step of predicting the predicted latency set of each scheduled task on the CPU container path and GPU slice path based on the pre-ready resource set and the computing power requirement level label, using a heterogeneous execution latency prediction model library, and downgrading tasks with insufficient prediction samples according to the computing power requirement level label to obtain a heterogeneous latency prediction result set, includes: taking the pre-ready resource set and the computing power requirement level label as input, extracting the corresponding algorithm type, input resolution, batch size, target carrier type, and current carrier occupancy rate for each AI task in the current scheduled batch to form a predictor input feature tuple; inputting the predictor input feature tuple into the heterogeneous execution latency prediction model library, wherein the heterogeneous execution latency prediction model library is obtained by gradient boosting regression fitting of pre-collected calibration samples, and outputting the first predicted latency of each AI task on the CPU container path and the latency on each available GPU slice path, respectively. The second predicted latency set on the path; for each AI task, it is determined whether the number of calibration samples corresponding to the predictor input feature tuple in the heterogeneous execution latency prediction model library is less than a preset sample number threshold. If it is less than the preset sample number threshold, a downgrade assignment is triggered. The AI ​​tasks with low computing power requirements are bound to the central processing unit container path, the AI ​​tasks with medium computing power requirements are bound to the graphics processor slice path with the middle remaining capacity in the pre-ready resource set, and the AI ​​tasks with high computing power requirements are bound to the graphics processor slice path with the largest remaining capacity in the pre-ready resource set. The assignment path of the AI ​​task is written back as the latency prediction result of the AI ​​task. If it is not less than the preset sample number threshold, the first predicted latency and the second predicted latency set are retained as the latency prediction result of the AI ​​task. The latency prediction results of all AI tasks are summarized, and the heterogeneous latency prediction result set is output.

[0018] Further, based on the heterogeneous latency prediction result set, a batch critical path minimization strategy with rank preference penalty and fragmentation awareness is adopted for task-carrier-slice allocation, and the corresponding containers are launched through the container orchestration engine to form a heterogeneous running container set. This includes: constructing an objective function for allocation decision using the heterogeneous latency prediction result set as input. The objective function is the weighted sum of the maximum of the cumulative latency of the CPU path and the cumulative latency of each GPU slice path, a rank preference penalty term, an internal slack term, and an external fragmentation term. The rank preference penalty term quantifies the degree to which the allocation path of all AI tasks in the current batch deviates from the computing power requirement level label; the internal slack term quantifies the difference between the allocated computing power of the GPU slice occupied by each AI task and the actual computing power consumed by the corresponding AI task; and the external fragmentation term quantifies the amount of space in the GPU remaining resources that cannot be pieced together to form an effective slice. Minimizing the objective function is the optimization objective. The algorithm employs a two-stage solution process: In the first stage, a greedy dispatcher completes the initial task-carrier-slice binding in milliseconds upon task arrival. In the second stage, an asynchronous background thread performs 2-opt local swaps on the binding results of a preset number of batches closest to the current optimization time, calculated after the swap, to determine the objective function value. The swap result is submitted only if the swapped objective function value is lower than the original objective function value, resulting in a task-carrier-slice binding decision set. Based on this decision set, the container orchestration engine issues container start / stop instructions and GPU slice activation instructions to the computing layer. For AI tasks bound to the CPU container path in the task-carrier-slice binding decision set, the corresponding CPU virtualization container is started. For AI tasks bound to the GPU slice path in the task-carrier-slice binding decision set, multi-process service-enabled algorithm containers are activated on the specified GPU slice, forming the heterogeneous runtime container set.

[0019] Further, the step of issuing container start / stop instructions and graphics processor slice activation instructions to the computing layer via the container orchestration engine based on the task-carrier-slice binding decision set, starting the corresponding CPU virtualization container for AI tasks bound to the CPU container path in the task-carrier-slice binding decision set, and activating the multi-process service enabled algorithm container on the specified graphics processor slice for AI tasks bound to the CPU slice path in the task-carrier-slice binding decision set, forming the heterogeneous running container set, includes: determining whether the graphics processor of the target computing node supports multi-instance graphics processor hardware partitioning capability based on the task-carrier-slice binding decision set; when the graphics processor of the target computing node supports multi-instance graphics processor hardware partitioning capability, proceeding according to the task-carrier-slice binding decision set... The slice specifications specified in the decision set are used to partition hardware-isolated slices using the multi-instance graphics processor (GPU) function. Multi-process services are activated on these hardware-isolated slices for time-dimensional process reuse, resulting in a hardware-isolated container instance. When the GPU of the target computing node does not support multi-instance GPU hardware partitioning capabilities, software slices are generated by the unified computing device architecture flow mechanism according to the slice specifications specified in the task-carrier-slice binding decision set. Multi-process services are activated on these software slices for time-dimensional process reuse, resulting in a software slice mode container instance. The hardware-isolated container instance or the software slice mode container instance is merged with the CPU virtualization container corresponding to the AI ​​task bound to the CPU container path in the task-carrier-slice binding decision set, forming the heterogeneous runtime container set.

[0020] Furthermore, the step of performing video stream inference based on the heterogeneous runtime container set, feeding back the measured latency after outlier filtering to the heterogeneous execution latency prediction model library, and outputting a standardized business event stream includes: taking the heterogeneous runtime container set as input, grouping real-time video stream slices from the perception layer by timestamp and pushing them into the inference task queue corresponding to each container, wherein the inference task queue bound to the central processing unit virtualization container adopts the first batch size, the inference task queue bound to the graphics processing unit slice container adopts the second batch size, and the second batch size is larger than the first batch size, thereby obtaining the inference task queue corresponding to each container;

[0021] Each container's algorithm inference engine performs object detection, behavior recognition, or multimodal fusion inference on video stream slices in the inference task queue. For each inference, the measured single inference latency is recorded, and a latency sample set is compiled. Frame-by-frame AI analysis results are output. A sliding window is applied to the latency sample set, and outlier samples exceeding three times the median absolute deviation are removed using the median absolute deviation method. The remaining latency samples are then fed back to the heterogeneous execution latency prediction model library for incremental updates to the predictor coefficients. The frame-by-frame AI analysis results are then structurally encapsulated via the application layer business gateway. This structural encapsulation includes adding camera identifiers, timestamps, event types, and confidence fields, and the standardized business event stream is output.

[0022] Furthermore, for the standardized business event flow and real-time power consumption, a limited-length real-time deficit queue and a limited-length energy consumption deficit queue are maintained. Through online solution with adaptive trade-off parameters and drift penalty, energy-saving action instructions are obtained. This includes: using the standardized business event flow and real-time power consumption collected in real-time by the computing layer as input, statistically analyzing the utilization rate of each container, event trigger frequency, number of active video channels, and instantaneous power consumption according to a preset sliding window period to form a real-time load-energy consumption profile; updating the limited-length real-time deficit queue and the limited-length energy consumption deficit queue based on the event processing latency and instantaneous power consumption in the real-time load-energy consumption profile, wherein the current value of the limited-length real-time deficit queue is the sum of the difference between the previous time slot value and the event processing latency exceeding the service level target threshold, constrained by an upper bound; the current value of the limited-length energy consumption deficit queue is the sum of the difference between the previous time slot value and the instantaneous power consumption exceeding the target power consumption upper limit, constrained by an upper bound, to obtain the updated limited-length energy consumption deficit queue. A long real-time deficit queue and an updated long-term energy consumption deficit queue are used. Based on the updated long-term real-time deficit queue and the updated long-term energy consumption deficit queue, a drift plus penalty target is constructed. The drift plus penalty target is the sum of the time-slot change of the sum of the squares of the updated long-term real-time deficit queue and the updated long-term energy consumption deficit queue multiplied by the instantaneous power consumption. The adaptive trade-off parameter is adjusted inversely with the current value of the updated long-term real-time deficit queue. The larger the current value of the updated long-term real-time deficit queue, the smaller the adaptive trade-off parameter. The smaller the current value of the updated long-term real-time deficit queue, the larger the adaptive trade-off parameter. The drift plus penalty target is obtained. Taking the minimization of the drift plus penalty target as the solution objective, block coordinate descent is used to decouple the decision into container start / stop sub-decision and graphics processor slice frequency or hibernation sub-decision. The energy-saving action command is output by iteratively solving the problem.

[0023] Further, the step of minimizing the drift plus penalty objective as the solution objective, using block coordinate descent to decouple the decision into container start / stop sub-decision and GPU slice frequency or sleep sub-decision, and iteratively solving to output the energy-saving action command, includes: taking the drift plus penalty objective as input, fixing the current value of the GPU slice frequency or sleep sub-decision, and solving the container start / stop sub-decision, wherein the decision space of the container start / stop sub-decision includes disabling redundant CPU containers, migrating the container corresponding to the medium computing power demand level task from the GPU slice to the CPU container, and only keeping the container corresponding to the basic recognition inference process on standby, to obtain the intermediate container start / stop decision; taking the intermediate container start / stop decision as input, fixing the intermediate container start / stop decision, and solving the GPU slice frequency or sleep sub-decision, wherein the GPU slice frequency or sleep sub-decision is determined by the block coordinate descent. The decision space for processor slice frequency or hibernation sub-decision includes reducing the frequency of a graphics processor slice to a preset idle level, merging tasks on multiple graphics processor slices with current carrier occupancy rates below a preset occupancy rate threshold into a single graphics processor slice, and setting the merged idle graphics processor slice to hibernation. Each frequency reduction or hibernation action is subject to a minimum duration constraint to suppress frequent switching jitter, resulting in an intermediate slice frequency decision. The intermediate container start / stop decision and the intermediate slice frequency decision are iterated alternately. In each iteration, the intermediate slice frequency decision is updated based on the current intermediate container start / stop decision, and the intermediate container start / stop decision is updated based on the current intermediate slice frequency decision, until the drift plus penalty target converges. The converged intermediate container start / stop decision and the intermediate slice frequency decision are merged and output to form the energy-saving action instruction.

[0024] The present invention has the following beneficial effects:

[0025] Firstly, by adopting a comprehensive computing power requirement quantification formula of "number of cameras × resolution × frame rate × algorithm complexity" and a three-level threshold classification method, the computing power requirement classification results are used as a unified trigger signal for container image pre-pull and slow-cycle reconfiguration of graphics processor space partitioning. This eliminates the cold start delay during dynamic allocation and hides the graphics processor reconfiguration overhead within the slow cycle of scene evaluation, thereby ensuring the task scheduling response speed in high-concurrency, multi-algorithm scenarios.

[0026] Secondly, by introducing a heterogeneous execution latency prediction model library to predict the execution latency of the CPU container path and each GPU slice path, and by adopting an intra-batch critical path minimization objective function with hierarchical preference penalty and dual fragmentation awareness, the CPU and GPU are upgraded from static binding to dynamic competitive matching based on predicted latency. This avoids cross-carrier mismatch and GPU internal idle waste, and the inference throughput is significantly better than the static binding scheme in more than 20 composite scenarios.

[0027] Third, by establishing a dual virtual queue of a real-time deficit queue and an energy consumption deficit queue with a sliding window limit, and designing the Lyapunov trade-off parameter as an adaptive function of the current value of the real-time deficit queue, the energy-saving decision-making is upgraded from time-based table plus threshold to online closed-loop control. It can adaptively cope with unexpected load fluctuations and achieve a power consumption reduction of more than 30% while ensuring the long-term real-time service level target constraint. This avoids the vicious cycle of service level target breach and subsequent conservative reservation caused by excessive energy saving. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of the overall process of a scenario-based AI task scheduling method for business halls based on computing power hierarchy according to the present invention.

[0029] Figure 2 This is a schematic diagram of the computing power classification assessment and resource pre-readiness sub-process of the present invention;

[0030] Figure 3 This is a schematic diagram of the heterogeneous latency prediction and dynamic computing power allocation sub-process of the present invention;

[0031] Figure 4 This is a schematic diagram of the online energy-saving closed-loop decision-making subprocess based on Lyapunov drift penalty according to the present invention. Detailed Implementation

[0032] Example 1

[0033] This embodiment describes a complete technical solution for a scenario-based AI task scheduling method for business halls based on computing power grading, covering the entire process from scenario feature acquisition, computing power grading assessment, heterogeneous latency prediction and assignment, inference execution and calibration, to online energy-saving closed-loop control.

[0034] like Figure 1 As shown, the AI ​​task scheduling method for business hall scenarios based on computing power tiers provided in this embodiment includes the following steps:

[0035] S1: Based on the video stream access list and algorithm subscription configuration list of the business hall, extract scene feature vectors and classify them according to computing power consumption calculation and preset thresholds to determine the computing power demand level label, and trigger the container image pre-pull and graphics processor space partition configuration of the corresponding level to obtain the pre-ready resource set.

[0036] The computing power tiering module obtains the video stream access list and algorithm subscription configuration list from the business hall management platform. The video stream access list includes the camera number, resolution (e.g., 1280×720, 1920×1080), and frame rate (e.g., 15 frames / second, 25 frames / second) for each camera. The algorithm subscription configuration list includes the subscribed algorithm types (e.g., customer flow statistics, face recognition, behavior recognition, etc.) and the call frequency of each algorithm. The scene feature extraction unit performs structured parsing on the above two lists, extracting a scene feature vector containing the number of cameras, the resolution of each camera, the frame rate of each camera, and the set of algorithm types. Each element of this vector reflects the hardware scale of the business hall, the video quality, and the subscribed algorithm types, respectively.

[0037] Based on the extracted scene feature vectors, each subscribed algorithm type is categorized into three complexity levels using a pre-defined algorithm complexity mapping table: basic recognition (e.g., passenger flow statistics), multimodal analysis (e.g., face recognition + behavior analysis), and high-precision composite scene (e.g., multi-object tracking + anomaly detection). The resolution, frame rate, and corresponding algorithm complexity coefficients are multiplied and weighted to obtain the overall computing power requirement. This overall computing power requirement is then compared with two pre-defined threshold levels to determine the computing power requirement level label.

[0038] Based on the computing power requirement level label, the level-resource mapping table is queried, and the corresponding resource configuration strategy is executed: For low computing power requirements, only the CPU container image containing basic recognition-type inference processes is pulled, and the GPU is divided into multiple fine-grained slice partitions or placed into sleep mode; for medium computing power requirements, both the CPU container image and the GPU slice container image are pulled, and the GPU is divided into mixed-specification slice partitions; for high computing power requirements, the GPU container image containing complete multimodal analysis and high-precision composite scene inference processes is pulled, and the GPU is divided into large slice partitions or configured as a dedicated GPU. The above pre-pulling and partitioning actions are executed immediately after the computing power grading assessment is completed in a slow cycle. Upon completion, a pre-ready resource set and its corresponding level label and GPU space partitioning scheme are output.

[0039] S2: Based on the pre-ready resource set and the computing power requirement level label, predict the predicted latency set of each scheduled task on the central processing unit container path and the graphics processing unit slice path through the heterogeneous execution latency prediction model library. Tasks with insufficient prediction samples are downgraded and assigned according to the computing power requirement level label to obtain a heterogeneous latency prediction result set.

[0040] After obtaining the pre-ready resource set, heterogeneous path latency prediction needs to be performed on each AI task in the current batch to be scheduled, in order to support subsequent optimal allocation decisions. The latency prediction unit extracts predictor input features for each AI task in the batch, including algorithm type, input resolution, batch size, target carrier type, and current carrier occupancy. The features are input into a heterogeneous execution latency prediction model library, which is trained offline using gradient boosting regression (such as XGBoost or LightGBM) on a small number of calibration samples. The model library outputs the predicted latency of each AI task on the CPU container path and the predicted latency set on each GPU slice path, respectively.

[0041] For each AI task, determine if the number of calibrated samples for the corresponding feature in the model library is less than a preset sample number threshold. If it is less than the threshold, trigger a degraded assignment protection: tasks with low computing power requirements are bound to the CPU container path; tasks with medium computing power requirements are bound to the GPU slice path with the middle remaining capacity in the pre-ready resource set; and tasks with high computing power requirements are bound to the GPU slice path with the largest remaining capacity. If the number of samples is not less than the threshold, retain the predicted latency of each path as the latency prediction result. Summarize the latency prediction results of all AI tasks to obtain a heterogeneous latency prediction result set.

[0042] S3: Based on the heterogeneous latency prediction result set, a batch critical path minimization strategy with hierarchical preference penalty and fragment awareness is adopted to perform task-carrier-slice assignment, and the corresponding container is launched by the container orchestration engine to form a heterogeneous running container set.

[0043] After obtaining the heterogeneous latency prediction result set, it is necessary to solve for the optimal task allocation scheme within the batch to minimize the total inference latency and control resource fragmentation. The task allocation module first needs to construct the objective function for allocation decision. Specifically, while simply minimizing the critical path within the batch can optimize inference latency, it does not consider the degree of matching between the allocation path and the computing power requirement level, nor does it consider the waste of underutilized computing power within the graphics processor slice and the fragmentation problem of the remaining space of the graphics processor being unable to form effective slices. Specifically, this invention introduces three penalty terms: a level preference penalty term to quantify the degree to which the allocation path deviates from the computing power requirement level label, an internal slack term to quantify the difference between the allocated computing power and the actual computing power consumed by the graphics processor slice, and an external fragmentation term to quantify the amount of remaining resources of the graphics processor that cannot be used to form effective slices. Thus, the following objective function is constructed:

[0044]

[0045] in, This represents the set of all AI tasks along the central processing unit container path. Represents a graphics processor slice The collection of all AI tasks. Indicates task Predictive inference latency on the CPU container path Indicates task In graphics processor slices Predictive inference latency, The maximum value between the cumulative latency of the CPU path and the cumulative latency of each GPU slice path is taken as the critical path within the batch. Penalty is the level preference penalty, InternalSlack is the internal slack item, and ExtFrag is the external fragmentation item. , , These are configurable weighting coefficients.

[0046] The solution employs a two-level mechanism: The first level, upon task arrival, uses a greedy dispatcher to complete the initial task-carrier-slice binding within milliseconds, ensuring real-time response; the second level, an asynchronous background thread performs 2-opt local swaps on the most recent batches at second-level intervals. When the objective function value after the swap is lower than before, the swap result is submitted to improve long-term decision quality. The container orchestration engine issues container start / stop instructions and GPU slice activation instructions to the computing layer according to the binding decision set, starting CPU virtualization containers or activating multi-process service-enabled algorithm containers on specified GPU slices, forming a heterogeneous set of running containers.

[0047] S4: Perform video stream inference based on the heterogeneous runtime container set, filter out outliers and feed the measured latency back to the heterogeneous execution latency prediction model library, and output a standardized business event stream.

[0048] After obtaining the heterogeneous set of runtime containers, the inference execution unit groups the real-time video stream slices from the perception layer by timestamp and pushes them into the inference task queues corresponding to each container. The inference task queues bound to the CPU virtualization containers are processed serially with a smaller first batch size; the inference task queues bound to the GPU slice containers are processed in parallel with a larger second batch size, making full use of the parallel computing capabilities of the GPU to obtain the inference task queues corresponding to each container.

[0049] Each container's algorithm inference engine performs object detection, behavior recognition, or multimodal fusion inference on video stream slices in the inference task queue. It records the measured single-inference latency for each inference attempt, compiling these into a latency sample set and outputting a frame-by-frame AI analysis result set. A sliding window is applied to the latency sample set, and the median absolute deviation method is used for outlier filtering: first, the median of the latency samples within the window is calculated; then, the absolute value of the difference between each sample and the median is calculated; finally, the median of these absolute values ​​is taken as the MAD value. Samples exceeding three times the MAD value are identified as outliers and removed. It should be noted that using the median absolute deviation method instead of the simple mean-standard deviation method effectively avoids statistical distortion caused by sudden abnormal loads. The filtered remaining latency samples are fed back to the heterogeneous execution latency prediction model library for incremental updates to the predictor coefficients, forming a stable prediction-execution-calibration closed loop.

[0050] The frame-by-frame AI analysis results are structured and encapsulated by the application layer business gateway, with the addition of camera identifier, timestamp, event type and confidence level fields, and a standardized business event stream is output.

[0051] S5: For the standardized business event flow and real-time power consumption, maintain a limited-length real-time deficit queue and a limited-length energy consumption deficit queue, and obtain energy-saving action instructions by online solution with adaptive trade-off parameters and drift penalty.

[0052] After obtaining the standardized business event flow, the energy-saving solution unit combines the power consumption data collected in real time by the computing layer to aggregate the data into a real-time load-energy consumption profile using a 5-minute sliding window. This profile includes the utilization rate of each container, the frequency of event triggers, the number of active video channels, and instantaneous power consumption. It should be noted that making immediate energy-saving decisions based solely on the latency and power consumption deviations of a given moment can easily lead to short-sighted decisions—significantly saving energy when latency is within the Service Limit of Default (SLO) at a certain moment may trigger subsequent continuous defaults; aggressively shutting down containers when power consumption is low at a certain moment may cause real-time performance to deteriorate in the next cycle. Therefore, this invention establishes virtual queues with sliding window limits for latency and power consumption respectively, incorporating the "whether historical defaults continue" into the decision-making criteria, thereby constraining energy-saving actions to not exceed the long-term SLO tolerance range.

[0053] Specifically, the update rule for the real-time deficit queue is as follows: the difference between the event processing latency of the current time slot and the Service Level Requirement (SLO) target is added to the queue value of the previous time slot, but an upper bound constraint is set to prevent infinite accumulation. The update rule for the energy consumption deficit queue is similar: the difference between the instantaneous power consumption of the current time slot and the target power consumption limit is added to the queue value of the previous time slot, also with an upper bound constraint. It should be noted that the purpose of setting the upper bound constraint is to prevent the infinite accumulation of historical defaults from causing long-term behavioral drift, thus gradually diminishing the impact of outdated default records on current decisions.

[0054] After obtaining the current value of the double deficit queue, a Lyapunov function is constructed as the decision objective. Specifically, the Lyapunov function takes half of the sum of squares of the double deficit queue to measure the current "deviation" of the system. The goal of energy-saving decision is to minimize the sum of single-slot Lyapunov drift and adaptive trade-off parameter multiplied by instantaneous power consumption. The adaptive trade-off parameter adjusts inversely with the real-time deficit queue: when the real-time deficit queue is large (real-time performance is close to default), the trade-off parameter automatically decreases and the energy-saving weight decreases; when the real-time deficit queue is small (real-time performance is sufficient), the trade-off parameter automatically increases and the energy-saving weight increases. This achieves adaptive energy-saving control without relying on an external timetable. Specifically, this design of the trade-off parameter adjusting inversely with the real-time deficit queue allows for automatic expansion of the energy-saving space and pursuit of lower power consumption when real-time performance is sufficient, while automatically tightening the energy-saving efforts and prioritizing SLO when real-time performance is tight, eliminating the core defect of preset timetables that cannot adapt to load changes. By decoupling the decision-making process into container start / stop sub-decision and graphics processor slice frequency or hibernation sub-decision through block coordinate descent, the energy-saving action command is obtained.

[0055] S6: According to the energy-saving action command, the computing power layer performs container start / stop and graphics processor slicing frequency or sleep adjustment to form the computing power state after energy saving, which serves as the feedback input for the scene feature vector extraction in the next cycle.

[0056] Upon receiving the energy-saving action command, the computing power scheduling controller performs fine-grained energy-saving operations on the computing power layer. These energy-saving operations include three types of atomic actions: The first type is container start / stop actions, where the controller sends a container stop command to the container orchestration engine to shut down redundant containers, or sends a container start command to start standby lightweight containers; the second type is slice frequency adjustment actions, where the controller sends a frequency adjustment command to the GPU driver to reduce the operating frequency of a specified slice from the working frequency range (e.g., 1.4GHz) to the idle frequency range (e.g., 0.9GHz), or from the idle frequency range to a sleep state; the third type is slice merging and sleep actions, where the controller migrates and merges inference tasks on multiple slices with occupancy rates below a preset threshold into a single slice, while the remaining vacant slices receive sleep commands and switch to low-power mode.

[0057] It should be noted that, to prevent system jitter caused by frequent switching of energy-saving actions, this invention sets minimum duration constraints for each type of energy-saving action. For example, a container must wait at least 5 minutes from startup to shutdown, a slice must remain in a state of frequency reduction and recovery for at least 3 minutes, and a slice must remain in a state of dormancy and activation for at least 10 minutes. This constraint effectively avoids energy-saving oscillations caused by instantaneous load fluctuations.

[0058] Meanwhile, the deficit queue update unit feeds back the actual operating status after energy saving to the dual deficit queues. The specific update process is as follows: The real-time deficit queue is updated based on the difference between the measured value of the event processing latency after energy saving and the SLO target—if the latency is lower than the SLO target, the queue value decreases; if it exceeds the SLO target, the queue value increases and an upper bound constraint is set to prevent infinite accumulation. The energy consumption deficit queue is updated based on the difference between the actual total power consumption after energy saving and the target power consumption upper limit—if the actual power consumption is lower than the target, the queue value decreases; if it exceeds the target, the queue value increases and an upper bound constraint is also set. The two updated deficit queues serve as inputs for the next time slot S5 energy-saving decision, providing historical default information for the next round of adaptive trade-offs.

[0059] Ultimately, the post-energy-saving computing power operation status is established, including the current number of containers running, the current operating frequency of each GPU slice, the current sleep state of each GPU slice, and the current value of the dual deficit queue. This post-energy-saving computing power status serves as feedback input for the feature vector extraction of the next scheduling cycle S1, enabling the computing power grading module to perceive the actual resource utilization of the previous cycle and thus adjust the pre-fetching and partitioning strategies. This forms a complete scheduling closed loop of "demand assessment and grading → dual-granularity heterogeneous allocation → inference execution calibration → Lyapunov energy-saving decision → computing power status feedback".

[0060] Example 2

[0061] Based on Example 1, this embodiment describes in detail the specific implementation method of extracting scene feature vectors from the video stream access list and algorithm subscription configuration list, completing computing power grading, and triggering resource pre-readiness.

[0062] like Figure 2 As shown, the process involves extracting scene feature vectors based on the video stream access list and algorithm subscription configuration list of the business hall, classifying them according to computing power consumption calculations and preset thresholds, determining computing power demand level labels, and triggering the pre-fetching of container images and the configuration of graphics processor space partitions for the corresponding level to obtain a pre-ready resource set, including:

[0063] S1.1: Perform structured parsing on the video stream access list of the business hall and the algorithm subscription configuration list, and extract scene feature vectors containing the number of cameras, resolution of each camera, frame rate of each camera, and set of subscription algorithm types.

[0064] The structured parsing unit in the computing power grading module acts as the front end for scene perception, responsible for pulling two types of raw configuration data from the business hall management platform. The first type is the video stream access list, which is exported by the business hall video surveillance system and is usually in JSON or XML format, containing configuration information for all connected cameras in the business hall. The second type is the algorithm subscription configuration list, which is exported by the AI ​​service platform and contains the intelligent analysis algorithms subscribed to by the business hall and their calling parameters.

[0065] The structured parsing unit first parses the video stream access list, extracting the configuration parameters of each camera from the list. Specifically extracted fields include: camera ID (used to uniquely identify each camera, such as CAM_001, CAM_002, etc.), resolution parameters (referring to the pixel resolution of the output video stream from that camera; common values ​​include 1280×720 for 720p, 1920×1080 for 1080p, 2560×1440 for 2K, 3840×2160 for 4K, etc.), and frame rate parameters (referring to the number of video frames output per second by that camera; common values ​​include 15 frames / second, 25 frames / second, 30 frames / second, etc.). It should be noted that different cameras within the same service hall may have different resolutions and frame rates. For example, the main hall may be equipped with 4K high frame rate cameras for facial recognition, while the warehouse area may be equipped with 720p low frame rate cameras used only for customer flow statistics.

[0066] Then, the structured parsing unit parses the algorithm subscription configuration list and extracts the set of subscribed algorithm types from it. The algorithm type field identifies the types of intelligent analysis algorithms that the branch has purchased, including but not limited to: customer flow statistics (used for counting the number of people entering and exiting and creating customer flow heatmaps), facial recognition (used for VIP customer identification and blacklist control), behavior recognition (used for abnormal behavior detection such as fall detection and loitering detection), license plate recognition (used for parking management), and object detection (used for lost object detection and missing item detection), etc. In addition, the algorithm subscription configuration list also contains the calling frequency parameters for each algorithm, that is, the calling strategy of the algorithm on each camera (such as calling every frame, calling once every 5 frames, etc.).

[0067] After parsing, the structured parsing unit integrates the above fields into a unified scene feature vector. The mathematical expression of this vector is: ,in Indicates the number of camera channels. Indicates the first The resolution of the road camera (expressed in total pixels, such as 1920×1080, which has a total of 2,073,600 pixels). Indicates the first The frame rate of the street camera This represents the set of algorithm types subscribed to by the business hall. This scenario feature vector serves as input for subsequent computational power requirements calculations, fully reflecting the expected hardware scale and algorithm complexity of the business hall's AI tasks.

[0068] S1.2: Based on the scene feature vector, each subscription algorithm type in the subscription algorithm type set is mapped to three complexity levels: basic recognition class, multimodal analysis class, and high-precision composite scene class through a preset algorithm complexity mapping table. The resolution of each path, the frame rate of each path, and the complexity level of the corresponding algorithm are multiplied and weighted to obtain the comprehensive computing power requirement value.

[0069] After receiving the scene feature vector output by the structured parsing unit, the computing power requirement calculation unit first queries a pre-set algorithm complexity mapping table. This mapping table is an algorithm complexity reference table pre-calibrated by this invention based on a large amount of actual test data, classifying common AI analysis algorithms in business halls into three complexity levels:

[0070] The first tier consists of basic recognition algorithms, including lightweight algorithms with single models and single tasks, such as passenger flow statistics (requiring only object detection and region counting), heatmap generation (based on object trajectory statistics), and simple motion detection. The computational complexity coefficient of this type of algorithm is... A value of 1 represents the baseline computational power consumption per unit pixel per unit time.

[0071] The second tier comprises multimodal analysis algorithms, including composite algorithms that fuse multiple models or involve multiple processing steps. Examples include face recognition (requiring face detection, key point localization, and feature comparison), behavior recognition (requiring object detection, behavior classification, and temporal analysis), and license plate recognition (requiring license plate detection, character segmentation, and OCR recognition). The computational complexity coefficient of this type of algorithm... The value is 3, which is 3 times that of the basic recognition class, because multimodal algorithms usually require multiple neural network models to be connected in series.

[0072] The third tier comprises high-precision composite scenario algorithms, including multi-target tracking, anomaly detection, and multimodal fusion, suitable for comprehensive smart service hall scenarios. The computational complexity coefficient of these algorithms is... The value is 5, which is 5 times that of the basic recognition class, because high-precision scenarios require a larger model backbone network, higher inference resolution, and multi-task joint optimization.

[0073] It should be noted that the determination of the above complexity coefficient is based on offline testing methods. Specifically, in a standard testing environment (e.g., equipped with an NVIDIA T4 graphics card and an Ubuntu 20.04 system), each type of algorithm was run at a uniform resolution (e.g., 1920×1080) and a uniform frame rate (e.g., 25 frames / second). The inference time per frame was measured, and then the ratio of this time value to the inference time per frame of a benchmark algorithm (e.g., passenger flow statistics) was taken as the complexity coefficient for that type of algorithm. This coefficient comprehensively reflects the relative differences in algorithm model size, inference computation load, and GPU memory usage.

[0074] After obtaining the complexity coefficients of each algorithm, the computing power requirement calculation unit calculates the computing power requirement for each camera channel, one by one. Taking the first... Taking a road camera as an example, if the road has subscribed to... If there are several algorithms, then the computing power requirement for this camera is:

[0075]

[0076] in, For the first The total number of pixels of the road cameras, For the first The frame rate of the street camera For the first The complexity coefficient of each algorithm, This is the sum of the complexity coefficients of all algorithms subscribed to on this path.

[0077] For example, let's say a certain business hall is equipped with one 1920×1080@25fps camera and subscribes to customer flow statistics ( ) and facial recognition ( Using the two algorithms, the computing power requirement for this camera is 1920×1080×25×(1+3)=207,360,000 pixels / second×4=829,440,000, which is approximately 830 million pixels / second.

[0078] By iterating through all the cameras in the service hall and summing their values, we can obtain the current comprehensive computing power requirement of the service hall. :

[0079]

[0080] This value comprehensively reflects the scale of the camera system. ), video quality ( and ) and algorithm complexity ( The impact of these three dimensions on computing power consumption will serve as the input basis for subsequent classification of computing power demand levels.

[0081] S1.3: Compare the comprehensive computing power requirement value with a preset first computing power threshold and a second computing power threshold, wherein the first computing power threshold is less than the second computing power threshold, the comprehensive computing power requirement value is determined to be low computing power requirement level when it is less than the first computing power threshold, medium computing power requirement level when it is between the first computing power threshold and the second computing power threshold, and high computing power requirement level when it is greater than the second computing power threshold, and output the computing power requirement level label.

[0082] The threshold classification unit compares the comprehensive computing power requirement value with the first threshold. Second threshold Compare the results and output the corresponding level labels. The threshold design follows these principles: First threshold Second threshold The value is determined based on the typical computing power configuration and cost-benefit analysis of the business hall. Taking a GPU server with 40GB of video memory as an example, its total computing power can support approximately [missing value]. Pixels per second (depending on GPU model and algorithm efficiency). First threshold. This threshold is typically set at 20% to 30% of the total computing power to differentiate scenarios with low computing power requirements (such as small business halls that only need to count customer traffic); the second threshold Typically set at 50% to 70% of the total computing power, this is used to differentiate between scenarios with medium computing power requirements (such as medium-sized business halls requiring facial recognition); the overall computing power requirement exceeds... It is then determined to be in the high computing power demand category (such as large business halls that require real-time analysis of multiple high-definition videos).

[0083] The rules for determining the level are: when When it is determined to be a low computing power requirement, the Tier_L level label is output; when When the requirement is determined to be medium computing power, the Tier_M level label is output; when... When this condition is met, it is determined to be in the high computing power demand category, and the Tier_H level label is output. Among these, This refers to the comprehensive computing power requirement calculated in step S1.2. For example, let's assume the comprehensive computing power requirement for a certain business hall is... = 500,000,000 (approximately 500 million pixels per second), first threshold =400,000,000, second threshold = 800,000,000. Because The business hall was determined to be in the medium computing power demand category (Tier_M), and the Tier_M level label was output.

[0084] S1.4: Query the level-resource mapping table according to the computing power requirement level label. The low computing power requirement level triggers the pre-fetching of the CPU container image containing only basic identification inference processes and divides the graphics processor into multiple slice partitions according to the first maximum video memory capacity or puts it into hibernation. The medium computing power requirement level triggers the joint pre-fetching of the CPU container image and the graphics processor slice container image and divides the graphics processor into mixed specification slice partitions according to the second maximum video memory capacity, wherein the second maximum video memory capacity is greater than the first maximum video memory capacity. The high computing power requirement level triggers the pre-fetching of the graphics processor container image containing complete multimodal analysis and high-precision composite scene inference processes and divides the graphics processor into slice partitions according to the third maximum video memory capacity or configures it as a dedicated whole card, wherein the third maximum video memory capacity is greater than the second maximum video memory capacity. The pre-ready resource set is obtained.

[0085] The resource pre-ready unit queries the level-resource mapping table based on the level label and executes the corresponding resource configuration strategy. The level-resource mapping table is a pre-defined resource configuration strategy reference table in this invention, which binds the computing power requirement level with the container image pre-pull strategy and GPU partition configuration.

[0086] For the low computing power requirement tier (Tier_L), the resource allocation strategy is as follows: First, container image pre-pull: only pull container images containing only basic recognition-type inference processes (such as the passenger flow statistics image pedestrian-detection:v1.0), and do not pull GPU container images. A typical configuration for pre-pulling CPU container images is: 1 to 2 containers, 2 to 4 CPU cores, and 2GB to 4GB of memory. Second, GPU configuration: divide the GPU into multiple fine-grained slice partitions with a first maximum video memory capacity (such as 1GB or 2GB) as the single slice limit. For example, a 40GB GPU can be divided into 20 1GB slices or 10 2GB slices for potential future high computing power needs; or, when the number of cameras is very small (such as less than 3), the GPU can be put into sleep mode to save static power consumption. It should be noted that the purpose of configuring the GPU partitions for sleep or fine-grained slicing in the low computing power requirement tier is to reserve expansion space for future load upgrades, rather than for immediate use.

[0087] For the medium computing power requirement tier (Tier_M), the resource allocation strategy is as follows: First, container images are pre-pulled together, simultaneously pulling CPU container images and GPU slice container images. The CPU container image pre-pulling configuration is: 2 to 4 containers, CPU quota of 4 to 8 cores, and memory quota of 4GB to 8GB. The GPU slice container image pre-pulling configuration is: 2 to 6 containers (determined based on the number of slices), GPU quota of 1 slice, and memory quota of 10GB to 20GB (determined based on slice specifications). Second, GPU partitioning divides the GPU into mixed-specification slice partitions based on the second maximum memory capacity (e.g., 10GB or 20GB). The configuration strategy for mixed-specification slicing is dynamically determined based on the estimated algorithm complexity: if face recognition is the primary function (requiring approximately 10GB of slicing), the configuration is 1GB.10GB + 2GB.20GB + 2GB.20GB; if behavior recognition is the primary function (requiring approximately 20GB of slicing), the configuration is 2GB.20GB + 1GB.10GB. The partitioning configuration for medium-performance computing needs balances computational resource utilization and multi-task concurrency capabilities.

[0088] For the high-computing-power requirement tier (Tier_H), the resource allocation strategy is as follows: First, container image pre-pull: Pull a GPU container image containing a complete multimodal analysis and high-precision composite scene inference process. A typical configuration for a high-computing-power GPU container image is: 4 to 8 containers, 1 large GPU slice or dedicated GPU, and 20GB to 40GB of memory. Second, GPU partitioning: Divide the GPU into large slice partitions or configure it as dedicated GPU based on the third maximum memory capacity (e.g., 40GB). Dedicated GPU configuration is suitable for high-precision composite scenarios, where all GPU resources (SM computing units, memory, bandwidth) are dedicated to a single task to ensure optimal inference performance. For example, on an A100 40GB GPU, it can be configured as a 4GB.40GB dedicated GPU, or divided into a combination of one 3GB.40GB slice and one 1GB.10GB slice.

[0089] The aforementioned pre-fetching and partitioning operations are performed immediately after the computing power grading assessment is completed, using a slow-cycle approach. The slow-cycle approach means that these configuration change operations are performed on a minute-by-minute rather than second-by-second basis, the purpose of which is to avoid frequent repartitioning operations affecting system stability. After pre-fetching and partitioning are completed, the resource pre-ready unit outputs a pre-ready resource set. This set contains the following information: a list of available CPU containers (container ID, image name, CPU quota, memory quota, current status), a list of available GPU slices (slice ID, slice specification, associated GPU, current status), and the current GPU partitioning configuration (partitioning mode, number of slices, resource quantity for each slice). The pre-ready resource set is then passed as output to the latency prediction module in step S2.

[0090] Example 3

[0091] Based on Example 2, this embodiment describes in detail the specific implementation of heterogeneous execution latency prediction, including predictor input feature construction, gradient boosting regression model inference, and cold start degradation assignment protection mechanism.

[0092] like Figure 3 As shown, based on the pre-ready resource set and the computing power requirement level label, the heterogeneous execution latency prediction model library predicts the predicted latency set of each scheduled task on the CPU container path and GPU slice path. Tasks with insufficient prediction samples are downgraded and assigned according to the computing power requirement level label, resulting in a heterogeneous latency prediction result set, including:

[0093] S2.1: Using the pre-ready resource set and the computing power requirement level label as input, for each AI task in the current batch to be scheduled, extract the corresponding algorithm type, input resolution, batch size, target carrier type and current carrier occupancy rate to form a predictor input feature tuple.

[0094] The heterogeneous latency prediction unit, as the core processing module of step S2, is responsible for constructing predictor input feature tuples for each AI task to be scheduled. This unit first receives the computing power requirement level label from step S1 and the pre-ready resource set from the pre-pull and partitioning unit, and then performs feature extraction on each AI task in the current batch to be scheduled.

[0095] A scheduled batch refers to a collection of newly arrived or changing AI inference tasks within the current scheduling period. Each AI task is triggered by a video stream slice, which is generated by the cameras in the perception layer at fixed time intervals (e.g., 1 second). A typical scheduled batch may contain dozens to hundreds of AI tasks, originating from different cameras and different algorithm subscription configurations. The heterogeneous latency prediction unit traverses each AI task within the batch, extracting feature information in the following five dimensions:

[0096] The first dimension is the algorithm type. This field identifies the type of inference algorithm executed for the AI ​​task and has the most significant impact on inference latency. The computational complexity of different algorithm types can differ by tens of times: taking passenger flow statistics algorithms as an example, they typically use lightweight object detection models (such as YOLOv5s), with a single-frame inference time of approximately 5 to 10 milliseconds; while high-precision face recognition algorithms require multiple levels of models such as face detection, key point localization, and feature extraction, with a single-frame inference time reaching 50 to 200 milliseconds. The value range of the algorithm type field corresponds to the list of algorithms registered on the business hall AI service platform, such as passenger flow statistics, face recognition, behavior recognition, license plate recognition, anomaly detection, and object recognition.

[0097] The second dimension is the input resolution. This field refers to the pixel size of the input image, which directly affects the computational load of inference. Pixel size and inference time have an approximately linear relationship: the inference time for a 1920×1080 resolution is approximately 2.25 times that of a 1280×720 resolution. Common resolutions in a business hall scenario include 720p (1280×720, approximately 920,000 pixels), 1080p (1920×1080, approximately 2.07 million pixels), 2K (2560×1440, approximately 3.68 million pixels), and 4K (3840×2160, approximately 8.29 million pixels). The input resolution field is read from the configuration parameters of the corresponding camera in the video stream access list.

[0098] The third dimension is batch size. This field indicates the number of images processed in a single inference run for this task. Batch size has a significant impact on inference efficiency: graphics processing units (GPUs) excel at batch parallel computing, and a larger batch size results in lower average inference latency per image; however, increasing the batch size also increases task queuing time, requiring a trade-off between throughput and latency. On the CPU path, due to the lack of large-scale parallel computing units, increasing the batch size has limited effect on latency improvement. The batch size field is dynamically set by the task queue management unit based on real-time load, and its value typically ranges from 1 to 32.

[0099] The fourth dimension is the target carrier type. This field identifies the execution path to which the task plan is assigned—whether it's a CPU container path or a GPU slice path. Before the assignment decision determines the target carrier type, the value of this field is determined by the cross-judgment between the computing power requirement level label in step S1 and the pre-ready resource set. If the level label is low computing power requirement and there are available CPU containers in the pre-ready resource set, then the target carrier type is a CPU container path; if the level label is medium or high computing power requirement, then the target carrier type is a GPU slice path. For GPU slice paths, the target slice number (e.g., GPU_Slice_1, GPU_Slice_3, etc.) must also be specified.

[0100] The fifth dimension is the current carrier utilization rate. This field indicates the percentage of currently allocated computing power to the total available computing power for the target execution path, reflecting the current workload status of the target carrier. Carrier utilization rate is an important contextual feature for prediction latency: when the carrier utilization rate is high (e.g., above 90%), new tasks need to wait in a longer queue, and the prediction latency should increase accordingly; when the carrier utilization rate is low (e.g., below 50%), new tasks can quickly obtain computing resources, and the prediction latency should decrease accordingly. The carrier utilization rate field is obtained in real time from the computing power layer monitoring system and refreshed every second.

[0101] The feature information from the above five dimensions is integrated into a unified feature tuple, mathematically expressed as: Here, algo_i represents the algorithm type, res_i represents the input resolution, batch_i represents the batch size, carrier_i represents the target carrier type, and load_i represents the current carrier occupancy rate. This feature tuple serves as the input for subsequent latency prediction, forming the predictor input feature tuple.

[0102] S2.2: Input the predictor input feature tuple into the heterogeneous execution latency prediction model library, wherein the heterogeneous execution latency prediction model library is obtained by gradient boosting regression fitting of pre-collected calibration samples, and outputs the first predicted latency of each AI task on the central processing unit container path and the second predicted latency on each available graphics processor slice path.

[0103] The heterogeneous latency prediction model library is the core prediction component of this invention, consisting of multiple gradient boosting regression models, each corresponding to a different execution path. This invention uses gradient boosting regression algorithms (such as XGBoost or LightGBM) as the basic algorithm for the prediction model. The offline training process of the heterogeneous execution latency prediction model library is as follows: First, calibration samples are pre-collected on the target computing power node. The collection of calibration samples covers all registered algorithm types, commonly used resolution configurations, and typical batch size combinations. For example, for the calibration scenario of the algorithm type "face recognition," multiple inference tests are performed at three resolutions (720p, 1080p, and 2K) and four batch sizes (1, 4, 8, and 16), recording the time consumed in each inference as the sample label. The data format of the calibration samples is... ,in Includes dimensions such as algorithm type, input resolution, and batch size. This corresponds to the measured inference latency. Taking the CPU container path as an example, a complete calibration process may generate hundreds of samples, covering the latency distribution of different algorithm types and configuration combinations.

[0104] Based on the collected calibration samples, independent gradient boosting regression models were trained for both the CPU container path and each GPU slice path. The CPU and GPU models have the same structure but different training data—the CPU model learns the latency patterns of the task along the CPU container path, while the GPU model learns the latency patterns along the GPU slice path. Cross-validation was used during training to determine the optimal combination of hyperparameters (such as the number of trees, maximum depth, and learning rate) to ensure the model's generalization ability.

[0105] During the online prediction phase, the heterogeneous latency prediction unit simultaneously inputs feature tuples into the CPU container path model and all GPU slice path models to obtain the corresponding predicted latency. Specifically, for the first [unit] within a batch... For each AI task, after extracting its feature tuples, the first prediction latency is obtained by querying the CPU model. And query each GPU slice model to obtain the second predicted latency set. ,in This represents the set of currently available GPU slices, and each element in the second prediction latency set corresponds to the prediction latency on a GPU slice.

[0106] It's important to note that the predicted latency of GPU slice paths varies between slices. Different GPU slices are physically located (in different compute units), and their memory bandwidth and compute frequency may differ slightly due to hardware aging or temperature variations. The heterogeneous latency prediction model library models each GPU slice individually, capturing these inter-slice differences. For example, if the memory bandwidth of a GPU slice is slightly lower than the baseline value during calibration, the predicted latency for that slice will increase accordingly; if a GPU slice has just completed a high-load task and is running at a lower frequency due to high temperature, the predicted latency for that slice will also increase accordingly. This refined slice-level prediction provides an accurate data foundation for the subsequent allocation optimization in the S3 step.

[0107] Summarize the predicted latency of all AI tasks in the batch across all execution paths, and output the first predicted latency set and the second predicted latency set. These sets serve as inputs for the subsequent S2.3 step degradation judgment and S3 step assignment optimization.

[0108] S2.3: For each AI task, determine whether the number of calibration samples corresponding to the predictor input feature tuple in the heterogeneous execution latency prediction model library is less than a preset sample number threshold. If it is less than the preset sample number threshold, trigger a downgrade assignment. The AI ​​tasks with low computing power requirements are bound to the CPU container path, the AI ​​tasks with medium computing power requirements are bound to the GPU slice path with the middle remaining capacity in the pre-ready resource set, and the AI ​​tasks with high computing power requirements are bound to the GPU slice path with the largest remaining capacity in the pre-ready resource set. The assignment path of the AI ​​task is written back as the latency prediction result of the AI ​​task. If it is not less than the preset sample number threshold, retain the first prediction latency and the second prediction latency set as the latency prediction result of the AI ​​task. Summarize the latency prediction results of all AI tasks and output the heterogeneous latency prediction result set.

[0109] The degradation protection unit checks the number of calibrated samples of each AI task's feature tuples in the model library. If the number of calibrated samples is less than a preset threshold (e.g., 20), a cold start degradation assignment is triggered. The unit executes preset assignment rules according to the computing power requirement level label: low-computing-power-requirement tasks are bound to the CPU container path; medium-computing-power-requirement tasks are bound to the GPU slice path with the middle remaining capacity in the pre-ready resource set; and high-computing-power-requirement tasks are bound to the GPU slice path with the largest remaining capacity. The assigned path is written back as the latency prediction result to prevent erroneous predictions due to insufficient model coverage from polluting the assignment decision. If the number of calibrated samples is not less than the threshold, the first and second prediction latency sets are retained as the latency prediction result. The latency prediction results of all AI tasks are summarized, and a heterogeneous latency prediction result set is output.

[0110] Example 4

[0111] Based on Example 1, this embodiment describes in detail the specific implementation of the batch critical path minimization assignment strategy with hierarchical preference penalty and fragment awareness, as well as the two-level solution mechanism.

[0112] Based on the heterogeneous latency prediction result set, a batch critical path minimization strategy with hierarchical preference penalty and fragment awareness is used to perform task-carrier-slice assignment, and the corresponding containers are launched through the container orchestration engine to form a heterogeneous runtime container set, including:

[0113] S3.1: Using the heterogeneous latency prediction result set as input, construct an objective function for the assignment decision. The objective function is the weighted sum of the maximum of the cumulative latency of the CPU path within the batch and the cumulative latency of each GPU slice path, a level preference penalty term, an internal slack term, and an external fragmentation term. The level preference penalty term quantifies the degree to which the assignment paths of all AI tasks in the current batch deviate from the computing power requirement level label. The internal slack term quantifies the difference between the allocated computing power of the GPU slice occupied by each AI task and the actual computing power consumed by the corresponding AI task. The external fragmentation term quantifies the amount of space in the GPU remaining resources that cannot be pieced together to form an effective slice.

[0114] The task assignment module takes the heterogeneous latency prediction result set as input and constructs the objective function for assignment decision. The design of this objective function comprehensively considers multiple optimization objectives of the assignment decision, including minimizing intra-batch critical path latency, penalizing the mismatch between computing power level and carrier type, reducing GPU resource waste, and controlling fragmentation rate.

[0115] The objective function is expressed mathematically as follows:

[0116] in, This represents the set of all AI tasks along the central processing unit container path. Represents a graphics processor slice The collection of all AI tasks. Indicates task Predictive inference latency on the CPU container path Indicates task In graphics processor slices Predictive inference latency, , , These are configurable weighting coefficients.

[0117] The first item is the critical path within the batch, which is the maximum of the cumulative latency of the CPU path and the cumulative latency of each GPU slice path. The critical path within the batch determines the overall completion time of the tasks in this batch. Minimizing this value ensures that the longest path within the batch is as short as possible. Suppose a batch contains 10 AI tasks, of which 6 tasks are assigned to the CPU path (cumulative latency 80 milliseconds), 4 tasks are assigned to GPU slice 1 (cumulative latency 60 milliseconds), and 2 tasks are assigned to GPU slice 2 (cumulative latency 55 milliseconds). Then the critical path within the batch is the maximum of 80 milliseconds and 60 milliseconds, i.e., 80 milliseconds.

[0118] The second term is Penalty, which is the sum of the deviations of all AI task assignment paths from their respective level labels. The motivation behind the Penalty is that assigning a task to a platform whose computational requirements are mismatched will result in additional performance overhead or resource waste. For example, assigning a low-computational-requirement task (such as passenger flow statistics) to a high-computational-requirement path (such as high-performance GPU slicing) may meet real-time requirements, but it wastes computational resources; assigning a high-computational-requirement task (such as high-precision face recognition) to a low-computational-requirement path (such as CPU containers) may lead to inference timeouts. The mathematical expression of the Penalty is:

[0119]

[0120] in, For the task The penalty weight is determined by the algorithm's complexity. This is an indicator function; it takes the value 0 when the dispatch carrier type matches the demand level, and 1 when they do not. Penalty weight. The value selection strategy is as follows: when low-computing-power-requirement tasks are assigned to the GPU path, Take a smaller positive value (e.g., 0.5), because this error only wastes resources but does not seriously affect performance; when high-computing-demand tasks are assigned to the CPU path, Take a positive value (e.g., 2.0), as this error could cause serious performance problems.

[0121] The third item is InternalSlack, which quantifies the wasted computing power of each task occupying but not fully utilizing a GPU slice. The mechanism for generating internal slack is: when the computing power of a GPU slice allocated to a task exceeds its actual required computing power, the difference between the two is the internal slack. For example, if a task actually requires 4GB of GPU memory but is allocated an 8GB slice, then 4GB of internal slack is generated. The mathematical expression for the internal slack item is:

[0122]

[0123] in, For the task The allocated GPU slice computing power (in units of video memory capacity). For the task The actual computing power required (determined based on algorithm type and input resolution). Minimizing internal slack items encourages the dispatcher to aggregate tasks with similar computing power requirements into the same slice, reducing computing power fragmentation.

[0124] The fourth item is the External Fragmentation item, ExtFrag, which quantifies the amount of remaining GPU resources that cannot be pieced together to form a valid slice. The mechanism for external fragmentation is as follows: when the total GPU memory minus the allocated memory results in less than the minimum slice size, this remaining memory is considered external fragmentation. For example, if the total GPU memory is 40GB and three 10GB slices (totaling 30GB) have been allocated, the remaining 10GB is exactly equal to the minimum slice size, so there is no external fragmentation. However, if three 8GB slices (totaling 24GB) have been allocated, the remaining 16GB cannot be divided evenly by the minimum slice size (10GB), resulting in 16GB mod 10GB = 6GB of external fragmentation. The mathematical expression for the External Fragmentation item is:

[0125]

[0126] in, Total GPU video memory. This represents the minimum memory capacity for a given slice. Minimizing external fragmentation prompts the allocator to consider memory alignment during allocation, reducing unused resources.

[0127] After obtaining the objective function for the assignment decision, the task assignment module passes it to the subsequent two-level solution unit for optimization.

[0128] S3.2: With minimizing the objective function as the optimization objective, a two-stage solution is performed: In the first stage, when a task arrives, the greedy dispatcher completes the initial task-carrier-slice binding in milliseconds; in the second stage, the background asynchronous thread performs 2-opt local swaps on the binding results of the number of pre-set batches closest to the current optimization time in seconds to calculate the objective function value after the swap. The swap result is submitted only when the objective function value after the swap is lower than the objective function value before the swap, thus obtaining the task-carrier-slice binding decision set.

[0129] The task assignment module employs a two-level solution mechanism to strike a balance between real-time response and decision quality. This mechanism is designed to address the dual constraints of AI task scheduling in the business hall: first, tasks require real-time arrival; excessively long assignment decisions will lead to task backlog and impact business response; second, the quality of assignment decisions has a significant impact on long-term system performance, and greedy strategies may fall into suboptimal solutions locally, necessitating a global optimization mechanism for improvement.

[0130] The first-level solution employs a millisecond-level greedy dispatch. The greedy dispatcher executes dispatch decisions immediately upon the arrival of each batch of tasks, keeping the time consumption within 10 milliseconds to ensure no blocking of real-time inference operations. The specific algorithm flow of the greedy dispatch is as follows: First, tasks to be dispatched are sorted by priority, determined by the urgency of the task's service level objective—tasks nearing SLO expiration have higher priority. Second, each task is traversed in priority order, enumerating all feasible execution paths (CPU paths and GPU slice paths) for that task. The objective function increment for that task on each path is calculated, and the path with the smallest increment is selected as the binding path for that task. Feasibility constraints include: GPU slices must not be over-dispatched (the allocation amount must not exceed the total slice capacity), and CPU containers must not be overloaded (the number of concurrent tasks must not exceed the container configuration limit). For example, suppose a batch contains 5 tasks (T1 to T5), the available GPU slices are Slice1 and Slice2, and CPU containers are available. The greedy dispatcher first processes T1 (highest priority), enumerating its objective function increments on CPU, Slice1, and Slice2 as 5, 3, and 4 respectively, and selects Slice1; then it processes T2, enumerating its increments on CPU, Slice1 (remaining capacity), and Slice2 as 4, 6, and 2 respectively, and selects Slice2; and so on, until all tasks are dispatched.

[0131] The limitation of the first-level greedy allocation is that it only considers the optimal solution for the current batch, without considering the co-optimization with the decisions of previous batches. For example, the optimal path for a task in the current batch may be Slice1, but from a global perspective, Slice1 is already nearly full, and high-priority tasks arriving in the future may not be allocated. The inherent flaw of the greedy strategy is its "short-sightedness," making it difficult to obtain the globally optimal solution.

[0132] The second-level solution involves second-level asynchronous 2-opt optimization. A background asynchronous optimization thread triggers every second (e.g., every 3 seconds) to attempt 2-opt local swaps on the binding results of the most recent batches (e.g., the last 5 batches) to improve long-term decision quality. 2-opt is a classic combinatorial optimization local search algorithm. Its basic idea is to randomly select two pairs of tasks to swap their assignment paths, determine whether the swap improves the objective function value, and only accept the swap if it does.

[0133] The detailed process of 2-opt swapping is as follows: First, randomly select two assigned task pairs from the historical binding results. and Among them, the task Assigned to path ,Task Assigned to path Then, try swapping the assignment paths of these two tasks, that is, let Assigned to , Assigned to Next, calculate the objective function value after the exchange. , and the objective function value before the exchange Compare; if If so, submit the exchange result and update the binding record; if If the original assignment remains unchanged, then the optimization thread will remain unchanged. To prevent the optimization thread from consuming excessive computing resources, only one round of 2-opt attempts (such as 50 random swaps) is performed each time it is triggered, instead of exhaustively searching all possible swap combinations.

[0134] The 2-opt optimization is performed asynchronously in the background, without blocking the real-time decision-making of the greedy dispatcher. After the optimization thread completes one round of 2-opt attempts, it submits the optimized binding decision to the task-carrier-slice binding decision set. Subsequent new batches of tasks will be dispatched based on the optimized binding decision, gradually improving the overall decision-making quality of the system.

[0135] It should be noted that the two-level solution mechanism resolves the conflict between real-time performance and decision quality through a dual-track strategy of "rapid response + continuous improvement". Greedy assignment ensures real-time response to business needs, while 2-opt optimization continuously corrects local defects of the greedy strategy in the background, enabling the system to maintain near-globally optimal assignment decisions in dynamically changing load environments.

[0136] S3.3: Based on the task-carrier-slice binding decision set, the container orchestration engine sends container start / stop instructions and graphics processor slice activation instructions to the computing layer. For AI tasks bound to the CPU container path in the task-carrier-slice binding decision set, the corresponding CPU virtualization container is started. For AI tasks bound to the CPU slice path in the task-carrier-slice binding decision set, the multi-process service enabled algorithm container is activated on the specified graphics processor slice, forming the heterogeneous running container set.

[0137] The container orchestration engine acts as a bridge between the task dispatch module and the computing infrastructure, responsible for translating binding decisions into executable hardware resource allocation instructions. After receiving the task-carrier-slice binding decision set, the computing power scheduler controller constructs corresponding container orchestration instructions based on the binding type of each decision, and distributes them to the container orchestration system of the computing power layer in the form of Kubernetes custom resources or DockerCompose service definitions.

[0138] For AI tasks bound to a CPU container path, the controller sends a container launch command to the container orchestration engine. This command includes the following key parameters: container image name (corresponding to the inference engine image of the executed algorithm, such as face-recognition:v2.1, pedestrian-detection:v1.5, etc.), CPU resource quota (set according to the computing power requirement level: 2 to 4 cores for low computing power, 4 to 8 cores for medium computing power, and 8 cores or more for high computing power), memory resource quota (usually 2GB to 16GB, depending on the size of the algorithm model), and network policy (allowing access to specified data sources and output targets). The CPU virtualization container is equipped with a lightweight inference engine, which is compiled and optimized for the CPU's SIMD instruction set (such as AVX2, AVX-512), and can fully utilize the multi-core parallel computing capabilities to execute inference tasks. Taking the passenger flow statistics algorithm as an example, this algorithm only requires object detection and region counting, with a relatively small computational load. Allocating 2 CPU cores and 2GB of memory is sufficient to meet the real-time requirements.

[0139] For AI tasks bound to a GPU slice path, the controller issues a GPU slice activation command to the container orchestration engine. This command includes the following key parameters: target slice number (e.g., GPU_Slice_2, GPU_Slice_5, etc.), slice resource specifications (determined by the GPU space partitioning scheme in step S1, such as 1g.10gb representing 1 GPU compute unit plus 10GB of VRAM, 2g.20gb representing 2 GPU compute units plus 20GB of VRAM), MPS service configuration (enabling multi-process service to support time-dimension multi-task multiplexing), and maximum concurrent connections (to prevent a single task from occupying too many MPS sessions and affecting other tasks). Algorithm containers activated on GPU slices run using Multi-Process Service (MPS) enabled mode. MPS is a GPU computing resource sharing mechanism provided by NVIDIA, allowing a single physical GPU to be divided into multiple compute slices, each slice running an independent process, with each process sharing GPU computing resources in a time-slice round-robin manner. Compared to the traditional container-exclusive GPU approach, MPS can significantly improve GPU utilization while maintaining a certain level of isolation, making it particularly suitable for the business characteristics of multi-task concurrency and dynamically changing computing power requirements in service hall scenarios.

[0140] It should be noted that the task allocation of the graphics processor slice path in this invention follows the principle of "slice reservation + dynamic reuse". Slice reservation means that when allocating a graphics processor slice for a task, a portion of the slice's computing resources (e.g., 20%) is reserved as spare capacity to cope with sudden loads from other tasks on the same slice. Dynamic reuse means that multiple AI tasks can be hosted on the same graphics processor slice simultaneously. Each task shares slice resources through an MPS session, and the MPS scheduler allocates computing time in a round-robin fashion according to time slices. This mechanism can both guarantee the service quality of each task and make full use of the idle computing power within the slice.

[0141] After the container orchestration engine executes the issued instructions, it waits for the containers to start and confirm their health status. Once all containers are in the Running state and the health checks have passed, it outputs a heterogeneous set of running containers. This set records each container's unique identifier (such as container ID, Pod name), the type of execution path it is bound to (CPU or GPU slice), the allocated hardware resource specifications, the container's current running status, and other information, which serves as input for the subsequent inference execution module.

[0142] Example 5

[0143] This embodiment, based on embodiment 4, describes in detail the adaptive container activation method that supports two modes: multi-instance graphics processor hardware partitioning and unified computing device architecture software slicing.

[0144] The process involves, based on the task-carrier-slice binding decision set, issuing container start / stop instructions and graphics processor slice activation instructions to the computing layer via the container orchestration engine. For AI tasks bound to the CPU container path in the task-carrier-slice binding decision set, the corresponding CPU virtualization container is started. For AI tasks bound to the CPU slice path in the task-carrier-slice binding decision set, multi-process service-enabled algorithm containers are activated on the specified graphics processor slice, forming the heterogeneous runtime container set, including:

[0145] S3.3.1: Based on the task-carrier-slice binding decision set, determine whether the graphics processor of the target computing node supports the hardware partitioning capability of multi-instance graphics processors.

[0146] The container orchestration engine first queries the graphics processor hardware capability profile of the target computing node to determine the GPU model and its support status for Multi-Instance Graphics Processor (MIG) hardware partitioning capabilities. Different NVIDIA GPU series support MIG to varying degrees: the A100 series with Ampere architecture and the H100 series with Hopper architecture support full MIG functionality; the T4 series with Turing architecture and the L4 series with Ada Lovelace architecture do not support MIG. For GPUs that support MIG, it is necessary to further query whether the current MIG mode is enabled—in some deployment scenarios, the MIG mode may be disabled due to driver version or system configuration reasons. This invention obtains the GPU's MIG support status and capability list by reading the device query interface of the NVIDIA Management Library (NVML).

[0147] S3.3.2: When the graphics processor of the target computing node supports the hardware partitioning capability of multi-instance graphics processors, the hardware isolation slice is drawn by the multi-instance graphics processor function according to the slice specification specified in the task-carrier-slice binding decision set, and the multi-process service is activated on the hardware isolation slice to perform time-dimensional process reuse, thereby obtaining a hardware isolation mode container instance.

[0148] When the GPU of the target computing node supports MIG, the container orchestration engine calls the NVIDIA MIG Manager interface to create the corresponding hardware isolation slice according to the slice specifications (such as 1g.10gb, 2g.20gb, 3g.40gb, etc.) specified by each task in the binding decision set. MIG hardware-isolated slices have the following characteristics: First, spatial isolation: each slice has an independent streaming multiprocessor (SM) computing unit, a dedicated memory partition, and an independent memory bandwidth controller. Resources between different slices do not interfere with each other, and load fluctuations in one slice will not affect the performance of other slices. Second, specification flexibility: MIG supports multiple slice specification combinations. For example, on an A100 with 40GB of memory, it can be configured as seven 1g.10gb slices (each with one SM unit + 10GB of memory) or two 3g.20gb slices (each with three SM units + 20GB of memory) + one 1g.10gb slice. Third, instance isolation: each MIG slice can be recognized by containers as an independent GPU device. Container runtimes (such as Docker NVIDIA Runtime) can directly assign MIG slices to specified containers without the need for an additional virtualization software layer.

[0149] Activating Multi-Process Service (MPS) on MIG slices enables time-dimensional process reuse. MPS is a GPU computing resource sharing mechanism provided by NVIDIA, allowing a single physical GPU or MIG slice to be further divided into multiple computing contexts. Each context shares the computing resources of the GPU / slice in a time-slice round-robin manner. Compared to traditional methods (such as CUDAContext switching), MPS has the following advantages: First, minimal context switching overhead. MPS eliminates kernel-level context switching latency by merging the CUDA flows of multiple processes into a single control flow. Second, enhanced resource isolation. MPS can limit the maximum GPU utilization percentage of each client process (measured as a percentage of CUDA operations), preventing a single process from excessively consuming resources. Third, increased throughput. MPS allows kernel functions from different processes to be executed interleaved in the time dimension, maximizing the utilization of GPU computing units.

[0150] To illustrate the MIG+MPS composite partitioning mechanism using specific numerical examples: Suppose a computing node in a business hall is equipped with one A100 40GB GPU. Currently, there are four AI tasks to be assigned (T1 to T4). The slice specifications specified by the binding decision set are 1GB / T2 each, and 1GB / T4 each. First, the container orchestration engine queries NVML to confirm that the GPU supports MIG, and calls MIGManager to create four 1GB / T4 hardware-isolated slices (SM_Instance_1 to SM_Instance_4). Then, MPS is enabled for each slice, and each slice is set to host a maximum of four MPS clients. Finally, the algorithm containers of each task are bound to the corresponding combination of MIG slice + T4 container (e.g., T1 container is bound to SM_Instance_1 + T4_VM_Client_1), resulting in four hardware-isolated container instances.

[0151] S3.3.3: When the graphics processor of the target computing node does not support the hardware partitioning capability of multi-instance graphics processors, a software slice is generated by the unified computing device architecture flow mechanism according to the slice specification specified in the task-carrier-slice binding decision set, and a multi-process service is activated on the software slice to perform time-dimensional process reuse, thereby obtaining a software slice mode container instance.

[0152] When the target computing node's GPU does not support MIG (such as consumer-grade or entry-level data center GPUs like T4 and L4), the container orchestration engine uses CUDA Stream software slicing to achieve time-division multiplexing of GPU resources. CUDA Stream is a mechanism in the CUDA programming model used to manage concurrent execution. Each Stream maintains an ordered queue of operations, and operations between different Streams can be executed in parallel or interleaved. This invention utilizes the following characteristics of CUDA Stream to achieve software slicing: First, computational resource partitioning: by allocating independent CUDA Streams to different tasks, GPU kernel execution and memory operations are grouped by Stream, forming logical isolation of computational resources; Second, priority scheduling: different Streams can be set with different priorities, and the kernels of high-priority Streams can preempt GPU computing units during the execution of kernels of low-priority Streams, achieving differentiated service quality; Third, memory bandwidth isolation: by limiting the memory access bandwidth of each Stream, it prevents a single task from excessively consuming memory bandwidth and affecting other tasks.

[0153] Activate MPS on CUDA Stream software slices for time-dimensional process reuse. Unlike MIG mode, MPS in software slice mode directly partitions the computation context on the physical GPU, rather than on the MIG slice. The specific implementation is as follows: First, create an MPS client process for each GPU task to be assigned and assign a unique MPS context ID (e.g., MPS_Context_1 to MPS_Context_4); then, configure resource limit parameters for each MPS client, including maximum GPU utilization (e.g., 25%), maximum kernel concurrency (e.g., 2), and memory allocation limit (e.g., 10GB); finally, bind the algorithm container of each task to the corresponding MPS context, and the MPS server schedules the computation requests of each client in a time-slice round-robin manner.

[0154] The isolation of the software slicing mode is weaker than that of the MIG hardware isolation mode—different tasks on the same physical GPU share all SM computing units and memory controllers, with excessive usage prevented only by the logical isolation mechanism of MPS. To compensate for this deficiency, this invention introduces the following compensation mechanisms in the software slicing mode: First, runtime monitoring: the container orchestration engine continuously monitors the resource usage of each MPS client. When it detects that a client is continuously exceeding its quota, a rate limiting mechanism is triggered. Second, dynamic adjustment: the resource quota of the MPS client is dynamically adjusted according to the real-time load. For example, when the load of a task suddenly increases, its quota is temporarily increased and the quotas of other tasks are reduced accordingly. Third, degradation protection: when GPU resources are exhausted, priority is given to ensuring resource supply for high-computing-demand tasks, and low-computing-demand tasks are migrated to CPU containers.

[0155] S3.3.4: Merge the hardware isolation mode container instance or the software slicing mode container instance with the CPU virtualization container corresponding to the AI ​​task bound to the CPU container path in the task-carrier-slice binding decision set to form the heterogeneous running container set.

[0156] After creating GPU container instances (whether in hardware isolation mode or software slicing mode), the container orchestration engine merges GPU container instances and CPU container instances into a unified heterogeneous runtime container set. The data structure of this heterogeneous runtime container set includes the following fields: a unique container identifier (e.g., Container_ID: "CPU_Container_001", "GPU_MIG_Instance_001", "GPU_CUDA_Stream_001", etc.), container type (CPU container, MIG hardware isolation instance, CUDA Stream software slice instance), a list of bound tasks (identifiers of all AI tasks running on this container), hardware resource configuration (number of CPU cores / memory size or MIG slice specification / CUDA Stream quota), and current running status (Pending / Running / Error). This heterogeneous runtime container set is then passed as output to the inference execution module in step S4.

[0157] Example 6

[0158] This embodiment, based on embodiment 1, describes in detail the specific implementation methods of video stream inference execution, latency sample outlier filtering and predictor online calibration, and business event structured encapsulation.

[0159] The process of performing video stream inference based on the heterogeneous runtime container set, filtering out outliers, and feeding back the measured latency to the heterogeneous execution latency prediction model library, outputting a standardized business event stream, includes:

[0160] S4.1: Taking the heterogeneous set of running containers as input, the real-time video stream slices from the perception layer are grouped by timestamp and pushed into the inference task queues corresponding to each container. The inference task queues bound to the central processing unit virtualization container adopt the first batch size, and the inference task queues bound to the graphics processing unit slice container adopt the second batch size. The second batch size is larger than the first batch size, thus obtaining the inference task queues corresponding to each container.

[0161] The task queue management unit, acting as the data access front-end for the inference execution module, is responsible for orderly distributing real-time video stream slices from the perception layer to the inference task queues of each container. The video stream from the perception layer is generated by each camera capturing images at fixed time intervals (e.g., 1 second), with each frame accompanied by a timestamp and metadata (camera ID, image resolution, etc.). Upon receiving video frames, the task queue management unit first groups them by timestamp—video frames from the same time period are aggregated into an inference batch, the batch size determined by system configuration parameters. The purpose of batch aggregation is to improve inference efficiency: for GPU containers, larger batches can fully utilize the massively parallel computing power of the GPU; for CPU containers, smaller batches can reduce the queuing time for a single inference attempt.

[0162] Differentiated batch size configuration is one of the key technical features of this invention. For inference task queues bound to a central processing unit virtualization container, a smaller first batch size (denoted as ) is used. The typical batch size is 4 to 8 frames. The reasons for using a smaller batch size for CPU containers are as follows: First, the parallel computing power of the CPU is limited, and a larger batch size would increase the completion time of a single inference iteration; second, CPU containers typically handle lightweight tasks (such as passenger flow statistics), with a single frame inference taking only 5 to 10 milliseconds, and a large batch size would lead to excessively high latency for the first frame; third, a smaller batch size facilitates real-time task scheduling, allowing high-priority tasks to be quickly inserted for execution. For inference task queues bound to GPU slice containers, a larger second batch size (denoted as...) is used. The typical value is 16 to 32 frames per second (FPS), with the specific value dynamically adjusted based on the slice's memory capacity. The reasons for using larger batch sizes for GPU containers are as follows: First, GPUs excel at large-scale parallel computing; the larger the batch size, the lower the average inference latency per unit image. Second, the GPU slice's memory capacity is a hard constraint; the batch size must ensure that all image data can be loaded into memory at once. Third, a larger batch size can increase the GPU's computational density and hide memory access latency.

[0163] Let's illustrate differentiated batch size configuration using specific numerical examples: Suppose a certain business hall is currently running 2 CPU containers (C1, C2) and 3 GPU slice instances (G1, G2, G3). Camera A (customer flow statistics) is bound to C1, configured... = 4 frames; Cameras B / C / D (face recognition) are bound to G1, configured. = 16 frames; Camera E / F (behavior recognition) is bound to G2, configured. =24 frames (this slice has a large video memory capacity); the camera G / H / I / J (multi-target tracking) is bound to G3, configured. = 20 frames. When camera A generates 30 frames per second, task queue C1 processes approximately 7.5 batches per second, with each batch consisting of 4 frames. When camera B generates 30 frames per second, task queue G1 processes approximately 1.875 batches per second, with each batch consisting of 16 frames per second.

[0164] S4.2: The algorithm inference engine in each container performs object detection, behavior recognition or multimodal fusion inference on the video stream slices in the inference task queue, records the measured single inference latency for each inference, summarizes them to form a latency sample set, and outputs frame-by-frame AI analysis results.

[0165] Each container's algorithm inference engine retrieves batch data from the inference task queue and executes the corresponding AI inference task. The algorithm inference engine is a software component that encapsulates the inference logic of a deep learning model, supporting multiple inference modes: For the CPU container, the algorithm inference engine executes basic recognition-type inference tasks. Typical tasks include: passenger flow statistics (counting the number of people in a specified area based on a lightweight object detection model), area occupancy detection (determining whether there are people in a specified area), and simple behavior detection (such as object detection, area intrusion detection, etc.). The inference engine used in the CPU container is compiled and optimized for the x86 architecture's SIMD instruction set (such as AVX2, AVX-512), utilizing multi-core parallelism to accelerate inference computation. Taking the YOLOv5s passenger flow statistics model as an example, the total processing time for 4 frames of images on a 4-core CPU is approximately 40 to 60 milliseconds, with a single-frame inference latency of approximately 10 to 15 milliseconds.

[0166] For GPU slice containers, the algorithm inference engine executes heavy inference tasks according to the batch size matched to the slice capacity. Typical tasks include: face recognition (concatenating face detection, key point localization, feature extraction, and other multi-level models), action recognition (action classification based on spatiotemporal convolution or Transformer), and multimodal fusion inference (fusion of video frames and multimodal perception of speech / text). The inference engine used by the GPU container supports CUDA acceleration and TensorRT optimization, which can fully leverage the massive parallel computing capabilities of the GPU. Taking a high-precision face recognition model as an example, the total time to process 16 frames of images on a GPU slice is approximately 80 to 120 milliseconds, and the single-frame inference latency is approximately 5 to 7.5 milliseconds.

[0167] After each inference is completed, the algorithm inference engine records the following latency information: First, the enqueue time. First, the timestamp when the batch enters the inference task queue; second, the start time. Third, the completion time. This refers to the timestamp when the inference process for that batch is completed and the results are output. The actual calculation method for single inference latency is as follows: (Time consumed by pure reasoning) and (Including end-to-end latency for queuing). This invention records two latency metrics simultaneously to support different analytical needs: pure inference latency is used to evaluate the performance of algorithms and hardware, while end-to-end latency is used to evaluate user experience and service quality.

[0168] The latency sample set aggregates inference latency records from all containers, forming a complete latency observation dataset. The frame-by-frame AI analysis result set contains the inference output for each frame of image, in the format {frame ID, list of detection boxes, list of classification labels, list of feature vectors}, which is then used by the business gateway for structured encapsulation.

[0169] S4.3: Apply a sliding window to the time delay sample set and use the median absolute deviation method to remove outlier samples within the window that exceed three times the median absolute deviation. Then, return the remaining time delay samples to the heterogeneous execution time delay prediction model library for incremental updates of the predictor coefficients.

[0170] The latency filtering unit detects outliers in the latency sample set to prevent abnormal samples from contaminating the predictor model. Outliers can be caused by factors such as inference timeouts due to GPU / CPU hardware failures, queuing congestion due to sudden increases in system load, and cold start delays caused by loading / unloading the inference model. Using these outliers directly to update the predictor without filtering will lead to model parameter shifts and reduce prediction accuracy.

[0171] This invention employs a sliding window method combined with the median absolute deviation (MAD) method for outlier detection. The specific steps are as follows:

[0172] The first step is to set up the sliding window. Set the sliding window length to... 100 samples (typically 100), window step size set to 100. 10 samples (typically 10), meaning each treatment A new sample slides the window once. Each time the window slides, the samples within it form a statistical unit.

[0173] The second step is to calculate the window statistics. Let the time delay samples within the window be... Calculate the sample median and the absolute value of the difference between each sample and the median. Take the median of these absolute values ​​as the MAD value: .

[0174] The third step is to identify outliers. The threshold for identifying outliers is... That is, when a sample satisfies If the sample is found to be an outlier, it is removed.

[0175] The fourth step is to backfeed valid samples. The filtered remaining latency samples are backfeeded to the heterogeneous execution latency prediction model library in an incremental update manner. The incremental update uses an online learning algorithm (such as stochastic gradient descent or online Bayesian update), which can adjust the model parameters without retraining.

[0176] The advantage of using the median absolute deviation method instead of the simple mean-standard deviation method is that the median is not sensitive to extreme values; even if there are many outliers within the window, the median can still reflect the central trend of the data well. The MAD value, as a robust measure of dispersion, is not affected by outliers, thus more accurately identifying true outliers. In contrast, the mean in the mean-standard deviation method is easily contaminated by outliers, leading to threshold shifts and an increased false positive rate.

[0177] S4.4: The frame-by-frame AI analysis results are structured and encapsulated by the application layer business gateway. The structured encapsulation includes adding camera identifier, timestamp, event type and confidence field, and outputting the standardized business event stream.

[0178] The business gateway unit performs structured encapsulation on the frame-by-frame AI analysis result set and outputs a standardized business event stream according to a unified business event format. The core of this structured encapsulation is converting heterogeneous inference output into a standardized format that is easy for downstream business systems to process, including the following four standard fields:

[0179] The first field is the camera identifier, denoted as Camera_ID. This field identifies the camera number or name that generated the analysis result, used to trace the source and geographical location of the event. The format of the camera identifier is "Business Hall Code_Area Number_Camera Number", for example, "BZ_01_003" represents Business Hall Code BZ, Area 01 (Business Hall Entrance), and Camera 003. The camera identifier is provided by the video stream metadata of the perception layer, and the algorithm inference engine retains this field when outputting the results.

[0180] The second field is the timestamp, denoted as Event_Timestamp. This field records the millisecond-level moment the event occurred, used for time series analysis, event correlation, and business statistics. The timestamp format follows the ISO 8601 standard, for example, "2026-04-20T10:30:15.234+08:00". The timestamp is provided by the video frame metadata of the perception layer, reflecting the image acquisition time rather than the inference completion time, to ensure the accuracy of the event timeline.

[0181] The third field is the event type, denoted as Event_Type. This field identifies the business semantics of the AI ​​inference output, used for business differentiation and classification statistics. The value of the event type depends on the type of AI task algorithm, and typical values ​​include: passenger flow statistics (PASSENGER_COUNT), face recognition (FACE_DETECTED, FACE_RECOGNIZED), behavior recognition (ABNORMAL_BEHAVIOR, LOITERING, TAKING_OBJECTS), multi-object tracking (MULTI_TARGET_TRACKED), etc. The granularity of the event type field can be configured: coarse-grained (e.g., "face detected") or fine-grained (e.g., "blacklisted personnel identified").

[0182] The fourth field is the confidence score, denoted as Confidence_Score. This field is the confidence score output by the inference model, ranging from 0 to 1, and is used to determine the credibility of an event. The confidence score is provided by the algorithm's inference engine when outputting the inference results, reflecting the model's certainty about the inference conclusion. For example, a confidence score of 0.95 for a face detection model means that the confidence that the detected bounding box actually contains a face is 95%; a confidence score of 0.82 for a face recognition model means that the confidence that the face matches a person in the database is 82%. Business systems can filter low-confidence events based on the confidence score threshold to reduce false positives.

[0183] The output format of the standardized business event flow is a JSON array.

[0184] Standardized business events are passed to the energy-saving solution module of the S5 step, and can also be subscribed to by other downstream applications such as the business management system and consumer behavior analysis system.

[0185] Example 7

[0186] Based on Example 2, this embodiment describes in detail the specific implementation methods of constructing a real-time load-energy consumption profile, maintaining a dual virtual deficit queue, constructing a Lyapunov drift plus penalty target, and solving for block coordinate descent.

[0187] like Figure 4 As shown, for the standardized business event flow and real-time power consumption, a limited-length real-time deficit queue and a limited-length energy consumption deficit queue are maintained. Through online solution using drift plus penalty with adaptive trade-off parameters, energy-saving action instructions are obtained, including:

[0188] S5.1: Using the standardized business event flow and the real-time power consumption collected by the computing power layer as input, the utilization rate of each container, the frequency of event triggering, the number of active video channels and the instantaneous power consumption are statistically analyzed according to the preset sliding window period to form a real-time load-energy consumption profile.

[0189] As the data acquisition front end for energy-saving decision-making, the load energy consumption statistics unit is responsible for collecting two types of raw data from the system and performing aggregation statistics to output a real-time load-energy consumption profile that reflects the current system operating status.

[0190] The first type of input data is a standardized business event stream, originating from the application-layer business gateway in step S4. The standardized business event stream is a sequence of business events output by the inference execution module. Each event record contains the following key fields: event timestamp (recording the completion time of the inference task), event type (identifying the AI ​​task type to which the event belongs, such as face recognition events, customer flow statistics events, etc.), container identifier (identifying the container ID that generated the event), and event processing latency (the total time taken for the inference task from queuing to completion, including queuing time and actual inference time). Taking a business hall as an example, the typical data format of a standardized business event stream is: {Timestamp: 2026-04-20 10:30:15.234, Type: Face Recognition, Container: CPU_Container_2, Latency: 45ms}.

[0191] The second type of input data is real-time power consumption data, which comes from the computing layer infrastructure monitoring interface. Real-time power consumption data reflects the current power consumption of the system and includes two sub-data: First, the real-time power consumption (in watts) of each CPU container, obtained through the resource monitoring interface provided by the container runtime. Each CPU container corresponds to one power consumption reading, typically ranging from 10W to 100W (depending on the number of CPU cores allocated to the container and the current load); Second, the real-time power consumption of each GPU slice, obtained through the NVIDIA Management Library (NVML) interface. Each GPU slice corresponds to one power consumption reading, and the power consumption range of a GPU slice is typically from 50W to 300W (depending on the number of computing units in the slice and the current computing load).

[0192] The load energy consumption statistics unit uses a sliding window mechanism to perform time-series aggregation on the above raw data. The specific configuration is as follows: the sliding window length is set to 5 minutes (300 seconds), and the window step size is set to 1 minute (60 seconds). This means that an aggregation calculation is triggered every 60 seconds, and each calculation covers the data within the most recent 5 minutes. The purpose of the sliding window mechanism is to smooth out instantaneous fluctuations and extract statistical features that reflect the system load trend.

[0193] The calculation method for aggregate statistics is as follows:

[0194] The first statistic is container utilization. Container utilization reflects the activity level of each container, calculated by dividing the cumulative core hours of each container in an active state (executing inference tasks) within a window by the total compute core hours. For example, if a CPU container is configured with 4 cores and is active for 120 seconds within a 5-minute window, then the container utilization within that window is 120 seconds / 300 seconds = 40%. GPU slice utilization is calculated similarly, showing the cumulative percentage of time the inference engine on each slice is in compute mode. It's important to note that container utilization is closely related to the number of containers: when there are too many containers, even with a low total load, the utilization of each container will be low, resulting in resource waste; when there are too few containers, the utilization rate will approach 100% during high-load periods, potentially leading to queuing and congestion. Container utilization is a key factor in S5.4's container start / stop decisions.

[0195] The second statistic is event trigger frequency. Event trigger frequency reflects the arrival rate of AI tasks and is calculated as the generation frequency of valid events (business events that successfully complete inference and have latency within the normal range) within a window, measured in events per second. For example, if a branch generates 6000 valid events within a 5-minute window, the event trigger frequency is 6000 / 300 = 20 events per second. Event trigger frequency is closely related to the branch's customer traffic: during peak hours (e.g., 10:00 AM to 12:00 PM), the event trigger frequency may reach over 50 events per second; during off-peak hours (e.g., 2:00 PM to 4:00 PM), the event trigger frequency may drop below 5 events per second. Event trigger frequency is an important indicator for predicting future load trends.

[0196] The third statistic is the number of active video feeds. This metric displays the number of cameras outputting inference tasks, reflecting the scale of currently running AI tasks. It's important to note that the number of active video feeds is not equal to the total number of connected cameras—some cameras may not be subscribed to the AI ​​algorithm at certain times, or may temporarily stop outputting due to changes in algorithm subscription configurations. The typical range for the number of active video feeds is 1 to 100 (depending on the size of the service center). The number of active video feeds serves as historical reference data for assessing the computing power requirements of the S1 step.

[0197] The fourth statistic is instantaneous power consumption. Instantaneous power consumption is the real-time power consumption reading at the end of the window, reflecting the total power consumption of the system at the current moment. Instantaneous power consumption is calculated as the sum of the power consumption of all CPU containers plus the sum of the power consumption of all GPU slices, measured in watts. For example, if a computing node is equipped with 2 CPU containers (power consumption of 30W and 45W respectively) and 3 GPU slices (power consumption of 150W, 180W, and 160W respectively), then the instantaneous power consumption is 30 + 45 + 150 + 180 + 160 = 565W. Instantaneous power consumption is the core input for updating the S5.2 energy deficit queue.

[0198] To avoid statistical distortion due to insufficient data during window initialization, the load energy consumption statistics unit uses an exponentially weighted moving average (EWMA) to smooth the initial statistical values. The EWMA calculation formula is as follows:

[0199]

[0200] in This is the current statistical estimate. This is the measured value for the current window. This is an estimate from the previous time slot. This is the smoothing coefficient (usually between 0.2 and 0.3). When the number of valid samples within the window is too small (such as when the system has just started), the EWMA mechanism gradually converges the statistical values ​​to a stable state, avoiding statistical abrupt changes caused by insufficient data.

[0201] The aggregated statistical results are integrated into a real-time load-energy consumption profile, with the data format being {window timestamp, list of container utilization, list of GPU slice utilization, event trigger frequency, number of active video channels, instantaneous power consumption}. This profile is then passed as output to the subsequent S5.2 virtual deficit queue update module.

[0202] S5.2: Based on the event processing latency and instantaneous power consumption in the real-time load-energy consumption profile, update the limited-length real-time deficit queue and the limited-length energy consumption deficit queue. The current value of the limited-length real-time deficit queue is the sum of the difference between the previous time slot value and the event processing latency exceeding the service level target threshold, and is constrained by an upper bound. The current value of the limited-length energy consumption deficit queue is the sum of the difference between the previous time slot value and the instantaneous power consumption exceeding the target power consumption upper limit, and is constrained by an upper bound. This yields the updated limited-length real-time deficit queue and the updated limited-length energy consumption deficit queue.

[0203] The virtual deficit queue maintenance unit updates the virtual deficit queues with two sliding window limits based on the event processing latency and instantaneous power consumption in the real-time load-energy consumption profile. The virtual deficit queue is the core data structure for energy-saving decision-making in this invention. It is used to accumulate historical default information reflecting both real-time and energy consumption dimensions of the system, providing a decision-making basis for subsequent Lyapunov optimization in S5.3.

[0204] This invention employs a virtual deficit queue design instead of a real task queue. The core motivation for this is that traditional queue systems struggle to simultaneously track default states in both real-time performance and energy consumption dimensions. Furthermore, the length of a real queue continuously increases as tasks arrive, making it difficult to set bounded constraints. The virtual deficit queue, through mathematical modeling, quantifies the deviations in real-time performance and energy consumption into scalar values. This not only reflects the accumulation of historical defaults but also prevents infinite accumulation through a sliding window mechanism.

[0205] The first queue is a limited-length real-time deficit queue, denoted as... This is used to accumulate the degree of default on event processing latency relative to the Service Level Objective (SLO). The update formula for the real-time deficit queue is:

[0206] in, This represents the real-time deficit queue value for the current time slot. The average event processing latency for the current time slot (extracted from the real-time load-energy profile). The service level target is typically set between 200 milliseconds and 500 milliseconds, depending on business needs. This is an upper bound constraint for the real-time deficit queue (usually set to 10 times the SLO).

[0207] The physical meaning of the real-time deficit queue is as follows: when the event processing latency exceeds the SLO (Solution Time Limit), the difference is added to the queue, indicating a certain default; when the event processing latency is below the SLO, the queue value decreases, indicating an improvement in the default status. The purpose of setting an upper bound constraint is to prevent the unlimited accumulation of historical defaults—even if the system remains in a default state, the queue value will not exceed [the limit]. This ensures that long-term decision-making behavior will not become unbalanced due to excessive accumulation of defaults.

[0208] For example, let SLO = 200ms. = 2000ms. Initial queue value. = 0. If the measured delay for a certain time slot is 250ms, then = max(0, 0 + 250 - 200, 2000) = 50ms; if the measured delay of the next time slot is 180ms, then = max(0, 50 + 180 - 200, 2000) = 30ms. As you can see, when the latency is improved, the queue value automatically decreases without any additional operation.

[0209] The second queue is the energy deficit queue with a limited lifespan, denoted as... This is used to accumulate the degree of default of instantaneous power consumption relative to the target power consumption ceiling. The update formula for the energy deficit queue is:

[0210]

[0211] in, This represents the energy deficit queue value for the current time slot. The instantaneous power consumption of the current time slot (extracted from the real-time load-energy consumption profile). Set the target power consumption limit (based on the energy consumption budget of the business hall, such as 600W). The upper bound constraint for the energy deficit queue (usually set as follows) (twice as much).

[0212] The physical meaning of the energy deficit queue is similar to that of the real-time deficit queue: when instantaneous power consumption exceeds the target upper limit, the difference is added to the queue; when instantaneous power consumption is lower than the target, the queue value decreases. The purpose of setting an upper bound constraint is also to prevent the unlimited accumulation of historical defaults.

[0213] Taking a specific numerical value as an example: Let = 600W, = 1200W. If the measured instantaneous power consumption in a certain time slot is 680W, then = max(0, 0 + 680 - 600, 1200) = 80W; If the measured instantaneous power consumption in the next time slot is 550W, then = max(0, 80 + 550 - 600, 1200) = 30W.

[0214] It should be noted that the updates to the two deficit queues are asynchronous. The real-time deficit queue is triggered by event processing latency (updated once after each batch of inference tasks is completed), while the energy consumption deficit queue is triggered by instantaneous power consumption sampling (sampled once per second). Although the update frequencies of the two queues differ, they ultimately converge to the Lyapunov optimization module in S5.3, which together determine the trade-off strategy for energy-saving actions.

[0215] The updated double deficit queue is passed as output to the subsequent S5.3 Lyapunov drift plus penalty target constructor module.

[0216] S5.3: Based on the updated time-limited real-time deficit queue and the updated time-limited energy consumption deficit queue, a drift plus penalty target is constructed. The drift plus penalty target is the sum of the time-slot change of the sum of the squares of the updated time-limited real-time deficit queue and the updated time-limited energy consumption deficit queue multiplied by the adaptive trade-off parameter and the instantaneous power consumption. The adaptive trade-off parameter is adjusted inversely with the current value of the updated time-limited real-time deficit queue. The larger the current value of the updated time-limited real-time deficit queue, the smaller the adaptive trade-off parameter. The smaller the current value of the updated time-limited real-time deficit queue, the larger the adaptive trade-off parameter. The drift plus penalty target is thus obtained.

[0217] The Lyapunov decision unit constructs a drift-penalty objective based on the updated double-deficit queue. Lyapunov optimization is a classic stochastic network control method. Its core idea is to measure the "deviation" of the system by constructing a Lyapunov function and to achieve long-term stability constraint optimization of the system by minimizing Lyapunov drift. This invention innovatively applies Lyapunov optimization to energy-saving decision-making in the field of computing power scheduling, minimizing long-term energy consumption while ensuring real-time SLO.

[0218] The Lyapunov decision unit's processing flow consists of three steps: first, calculating the Lyapunov function; second, calculating the Lyapunov drift; and third, constructing a drift-penalized objective.

[0219] The first step is to calculate the Lyapunov function. The Lyapunov function is constructed using half the sum of squares of the double deficit queues, mathematically expressed as:

[0220]

[0221] The physical meaning of the Lyapunov function is to measure the degree to which a system deviates from its ideal state. When the twin deficit queue value is large, the Lyapunov function value is large, indicating that the system is in an "unbalanced" state and measures need to be taken to restore balance; when the twin deficit queue value is small, the Lyapunov function value is small, indicating that the system is in a "healthy" state. For a specific numerical example: Let... = 500ms, = 200W, then = (500² + 200²) / 2 = (250000 + 40000) / 2 = 145000.

[0222] The second step is to calculate the Lyapunov drift. The Lyapunov drift is defined as the difference between the Lyapunov function values ​​of two adjacent time slots, mathematically expressed as:

[0223]

[0224] The physical meaning of Lyapunov drift is the changing trend of the degree of deviation of the system. When When the system tends to reach equilibrium; when At this point, the system tends to become unbalanced. The goal of minimizing Lyapunov drift is to bring the system to equilibrium, even as the double deficit queue value gradually decreases.

[0225] The third step is to construct the drift-plus-penalty objective. The mathematical expression for the drift-plus-penalty objective is:

[0226]

[0227] in, For Lyapunov drift, For adaptive trade-off parameters, This refers to instantaneous power consumption. The physical meaning of the drift-penalty objective is to minimize the system imbalance while considering the impact of instantaneous power consumption. Adaptive trade-off parameters. Used to balance real-time performance with energy efficiency. The larger the value, the higher the energy-saving weight; The smaller the value, the higher the weight of real-time performance guarantee.

[0228] This invention designs a dynamic adjustment mechanism for adaptive trade-off parameters. The core idea is: the larger the real-time deficit queue, the closer the system's real-time performance is to default; in this case, the energy-saving weight should be reduced to prioritize real-time performance. Conversely, the smaller the real-time deficit queue, the more ample the system's real-time performance; in this case, the energy-saving weight can be increased to pursue lower power consumption. The calculation formula for the adaptive trade-off parameters is:

[0229]

[0230] in, and These represent the minimum and maximum values ​​of the trade-off parameters (prioritizing real-time performance) and (allowing for deep energy savings), respectively. This is the adjustment coefficient. Taking a specific value as an example: Let... = 0.5, = 2.0, = 0.01. When =0 (sufficient real-time performance). = 0.5 + (2.0 - 0.5) × 1 / (1 + 0) = 2.0, the energy-saving weight is the highest; when = 1000ms (real-time approaching default). = 0.5 + (2.0 - 0.5) × 1 / (1 + 10) ≈0.64, the energy saving weight is significantly reduced.

[0231] It should be noted that the adaptive trade-off mechanism of this invention does not rely on external timetables (such as work schedules, holiday schedules, etc.) and can automatically adjust energy-saving efforts according to actual load conditions. When the load on the sales hall suddenly increases (such as during promotional activities), the real-time deficit queue automatically increases, the trade-off parameter automatically decreases, and the energy-saving weight decreases, prioritizing ensuring real-time SLO (Solution Time Limit). When the load on the sales hall decreases, the real-time deficit queue automatically decreases, the trade-off parameter automatically increases, and the energy-saving weight increases, pursuing lower power consumption. This adaptive mechanism, which requires no manual intervention, overcomes the core defect of traditional preset timetables that cannot respond to dynamic load changes.

[0232] The drift plus penalty target is output and passed to the subsequent S5.4 block coordinate descent solution module.

[0233] S5.4: Taking the minimization of the drift plus penalty objective as the solution objective, block coordinate descent is used to decouple the decision into container start / stop sub-decision and graphics processor slice frequency or hibernation sub-decision, and the energy-saving action command is output through iterative solution.

[0234] The block coordinate descent solving unit decouples the energy-saving decision problem into two sub-problems that are solved alternately: the first sub-problem is the container start / stop sub-decision, with the decision variable being the start / stop state of each container; the second sub-problem is the GPU slice frequency or hibernation sub-decision, with the decision variable being the operating frequency level of each GPU slice. The specific alternating iteration steps and minimum duration constraints are described in detail in Example 8. After convergence, the unit outputs energy-saving action instructions containing a list of container start / stop instructions and a list of GPU slice frequency instructions, which are then passed to the execution module in step S6.

[0235] Example 8

[0236] Based on Example 7, this embodiment describes in detail the alternating iterative steps of the block coordinate descent decoupling solution and the specific implementation of the minimum duration constraint.

[0237] The goal is to minimize the drift plus penalty objective. Block coordinate descent is used to decouple the decision into container start / stop sub-decision and GPU slice frequency or hibernation sub-decision. The energy-saving action command is iteratively solved and output, including:

[0238] S5.4.1: Using the drift plus penalty target as input, fix the current value of the graphics processor slice frequency or the sleep sub-decision, and solve the container start-stop sub-decision. The decision space of the container start-stop sub-decision includes disabling redundant CPU containers, migrating the container corresponding to the medium computing power demand level task from the graphics processor slice to the CPU container, and keeping only the container corresponding to the basic recognition inference process on standby, to obtain the intermediate decision of container start-stop.

[0239] In this embodiment, given a fixed GPU slicing decision, the start / stop state of each container is optimized. The decision space for the container start / stop sub-decision includes three basic operation modes:

[0240] The first operation is to disable redundant containers. When the utilization of a CPU container is consistently low (e.g., the average utilization over the last three sliding windows is below 30%), it indicates that the container's computing power configuration exceeds the current load demand, and some containers can be shut down to save power. Disabling redundant containers requires meeting the following feasibility constraints: First, all AI tasks running on the container must be successfully migrated to other available containers, and no task loss or timeout should occur during the migration process; second, newly enabled standby containers (e.g., standby containers that are currently closed) require time to go from cold start to readiness. (Typically 20 to 60 seconds), during which time the system needs to ensure sufficient redundancy capacity to handle sudden loads. Therefore, the decision to disable redundant containers should follow the "start first, then stop" principle: first activate the backup container and confirm its readiness, then shut down the target container to ensure uninterrupted service. For example, suppose there are currently 4 CPU containers (C1 to C4), and the utilization rate of C4 is consistently low (20%). Tasks running on C4 can be migrated to C1 to C3 (whose current utilization rates are 65%, 70%, and 55%, respectively). The decision is to shut down C4 and evenly migrate the tasks on C4 to C1 to C3.

[0241] The second operation is inter-container task migration. When the utilization of a GPU slice is too high (e.g., above 85%) or too low (e.g., below 20%), some tasks can be migrated from the GPU slice to a CPU container (or vice versa) to balance the utilization of heterogeneous resources. The decision to migrate tasks must meet the following feasibility constraints: first, migration overhead constraints, i.e., the inference latency of the migrated task must not exceed 1.2 times its SLO; second, carrier capacity constraints, i.e., the utilization of the target carrier must not exceed 95%. The specific migration logic is as follows: when the utilization of a GPU slice is too high, prioritize migrating out the tasks with the lowest computing power requirement level on that slice (e.g., basic recognition tasks); when the utilization of a GPU slice is too low, evaluate whether migrating the tasks on it to a CPU container satisfies the SLO constraint—if it does, then execute the migration and reduce the frequency of the GPU slice or put it to sleep. Inter-container task migration is only performed on tasks with medium computing power requirements because these tasks are less sensitive to carrier type.

[0242] The third approach is container streamlining. This is used when the system is under extremely low load (e.g., the frequency of event triggering is below a threshold). The typical value is 2 events / second, and the duration exceeds [a certain value]. When the time limit is 5 minutes (typically), all containers except for basic identification containers can be shut down. Container streamlining must meet the following feasibility constraints: First, a minimum number of containers must be kept alive, i.e., at least [number of containers must be retained]. There are two main constraints: first, a single container is used to handle unexpected events; second, there are basic service retention constraints, meaning that containers corresponding to basic identification algorithms such as passenger flow statistics must not be shut down. After simplification, the system enters a low-power standby mode, retaining only a very small amount of computing resources to maintain core monitoring functions.

[0243] Under the drift-penalty objective, the above decision space is solved using greedy enumeration or branch-and-bound heuristic search. The greedy enumeration strategy is as follows: sort each operation in descending order of its marginal improvement to the objective function, try each operation in turn and check the feasibility constraints, and terminate when all feasible operations have been tried or the maximum number of enumerations has been reached, and output the intermediate decision of starting and stopping the container.

[0244] S5.4.2: Taking the container start / stop intermediate decision as input, fix the container start / stop intermediate decision, and solve the graphics processor slice frequency or hibernation sub-decision. The decision space of the graphics processor slice frequency or hibernation sub-decision includes reducing the frequency of the graphics processor slice to a preset idle level, merging the tasks on multiple graphics processor slices with the current carrier occupancy rate lower than the preset occupancy rate threshold into a single graphics processor slice, and setting the merged idle graphics processor slice to hibernation. Each frequency reduction or hibernation action is set with a minimum duration constraint to suppress frequent switching jitter, and the slice frequency intermediate decision is obtained.

[0245] In this embodiment, under the premise of fixed container decisions, the operating frequency level of each GPU slice is optimized. The decision space of GPU slice frequency or hibernation sub-decision includes two basic operating modes:

[0246] The first operation is slice frequency adjustment. The power consumption of a GPU slice is approximately linearly related to its operating frequency: when the frequency drops from the normal level (e.g., 1.4GHz) to the idle level (e.g., 0.9GHz), dynamic power consumption can be reduced by approximately 40% to 50%. Frequency adjustment decisions must meet the following feasibility constraints: first, load feasibility constraint, meaning the inference latency of all tasks on the slice after frequency reduction must not exceed its SLO; second, frequency level constraint, meaning the target frequency must be a preset discrete level (e.g., 1.4GHz, 1.2GHz, 0.9GHz, 0.7GHz, etc.). Frequency adjustment employs a predictive correction mechanism: first, the load level of the next time slot is predicted by extrapolating the trend of event trigger frequency. Then, calculate whether the expected latency after frequency reduction meets the SLO constraint; finally, if the predicted load will decrease (e.g., the business hall is about to enter the off-peak period) and the latency after frequency reduction meets the SLO, then frequency reduction is executed; otherwise, the current frequency is maintained.

[0247] The second operation is slice merging and hibernation. This occurs when the utilization of multiple GPU slices is low (e.g., below 25%) and the duration exceeds [a certain value]. (Typical value is 3 minutes) When the time is right, tasks on these slices can be merged into at least a few slices, and the empty slices are switched to sleep mode to eliminate static power consumption. The power consumption of GPU slices in sleep mode is close to zero (only minimal sustaining circuitry is retained), which can save an additional 30% of power consumption compared to idle frequency. The decision to merge slices into sleep mode must meet the following feasibility constraints: first, the utilization rate of the target slice after merging must not exceed 95%; second, the delay of the sleep slice reactivating. The recovery time from hibernation to normal operation must not exceed the maximum acceptable waiting time for the system (usually 10 seconds); thirdly, the number of hibernation slices must not exceed 50% of the total number of slices to ensure that the system retains sufficient redundancy capacity.

[0248] Minimum duration constraints are a key mechanism to prevent frequent switching jitter.

[0249] This invention maintains a timer for each frequency level state of each GPU slice, denoted as . Where j represents the slice number and k represents the frequency level number (e.g., 0 represents sleep, 1 represents the lowest frequency level, 2 represents the idle level, and 3 represents the normal frequency level). The timer update rule is as follows: When a slice is decided to switch to level k, first check if the slice has just switched from level k'—if it switched from level k' to level k, and level k' ≠ level k, then reset. Set the timer to 0 and stop counting for gear k'; if switching from gear k to gear k (i.e., a re-decision within the same gear), then keep the timer active. Unchanged. Only when Reaching minimum duration (Typical value is 30 seconds) before shifting from gear K to other gears is allowed.

[0250] Let's illustrate the timer mechanism with a specific numerical example: Suppose a GPU slice is currently at its normal frequency level (level 3), and the timer value is 100 seconds. If, at T=100 seconds, load forecasting indicates a future load decrease, the dispatcher decides to downclock the slice to the idle level (level 2). At this time, the timer value for level 2 is 0, and the timer for level 3 stops. If, at T=120 seconds, load forecasting indicates a load rebound, the dispatcher decides to raise the slice back to its normal frequency level (level 3). Since the timer for level 3 has reached its limit... =30 seconds, frequency ramping is allowed. If, at T=45 seconds, load forecasting indicates a load rebound, the dispatcher will decide to ramp the slice back to the normal frequency level. Because the timer for level 3 is only 45 seconds (less than...) =30-second tolerance limit), frequency upscaling is blocked, and slices remain idle. This mechanism effectively prevents frequent frequency switching caused by short-term load fluctuations.

[0251] S5.4.3: The intermediate decisions for container start / stop and slice frequency are iterated alternately. In each iteration, the intermediate decisions for slice frequency are updated based on the current intermediate decisions for container start / stop and the intermediate decisions for container start / stop are updated based on the current intermediate decisions for slice frequency, until the drift plus penalty target converges. The converged intermediate decisions for container start / stop and slice frequency are merged and output to form the energy-saving action command.

[0252] In this embodiment, alternating iterations are performed between container start / stop decisions and GPU slicing decisions to seek a joint optimal solution that combines drift and penalty objectives. The core idea of ​​alternating iteration is that changes in container decisions affect the load distribution of GPU slices, thus affecting the optimal GPU slice frequency decision; changes in GPU slice decisions also affect the load allocation decision of containers. Therefore, multiple rounds of alternating optimization are needed to gradually approach the joint optimal solution.

[0253] The execution flow of the alternating iterative control unit includes the following steps:

[0254] The first step is initialization. Set the maximum number of iterations. (Typical value is 5) and convergence threshold (Typical value is 0.001). Initialize the container start / stop intermediate decision to the current running state, and initialize the GPU slice frequency intermediate decision to the normal frequency range. Calculate the initial drift plus penalty target value. .

[0255] The second step is to fix container decisions and optimize GPU slicing decisions. This involves refining intermediate decisions regarding container start-up and shutdown. Fixed, GPU slice frequency decision space under drift plus penalty objective Perform the solution. The specific optimization method is the same as step S5.4.2, while applying a timer constraint (only allowing when the Timer has been reached). (gear shifting), to obtain the intermediate decision of the new GPU slice frequency. .

[0256] The third step is to fix container decisions and optimize GPU slicing decisions. This involves refining intermediate decisions regarding container startup and shutdown. Fixed, GPU slice frequency decision space under drift plus penalty objective Perform the solution. The specific optimization method is the same as step S5.4.2, while applying a timer constraint (only allowing when the Timer has been reached). (gear shifting), to obtain the intermediate decision of the new GPU slice frequency. .

[0257] Step 4: Check convergence. Calculate the drift penalty target value for the (k+1)th iteration. Compare this value with the value from the k-th round. If If the convergence condition is met, the iteration stops; otherwise, if the convergence condition is not met and k+1 < If k = k+1, then return to step one and continue iterating; if k+1 ≥ If the maximum number of iterations is reached, the iteration will stop.

[0258] There are two types of collaborative constraints that require special attention during the alternating iteration process:

[0259] The first type of collaborative constraint is container adjustments triggered by changes in GPU slice load. If a GPU slice is decided to downclock or go to sleep during iteration, tasks on that slice will be migrated to CPU containers, causing an increase in the load on the CPU containers. At this time, if the utilization of the CPU containers exceeds a safety threshold (e.g., 85%), container expansion (i.e., reopening closed standby containers) needs to be triggered to avoid SLO (Solution Time Limit) default. Similarly, if a GPU slice is decided to upclock, it can be evaluated whether some tasks on CPU containers can be migrated back to the GPU slice to free up CPU resources.

[0260] The second type of collaborative constraint involves slice adjustments triggered by container state changes. If a CPU container is shut down during iteration, tasks on that container will migrate to GPU slices, causing an increase in the load on those GPU slices. In this case, it's necessary to reassess whether the GPU slice's frequency level still meets the SLO constraint—if not, the frequency needs to be increased or a dormant slice needs to be woken up. Similarly, if a CPU container is enabled (switching from a closed to an open state), the load on the GPU slice decreases, and it's possible to reassess whether the frequency can be decreased or some GPU slices can be put to sleep.

[0261] Through iterative adjustments to the aforementioned collaborative constraints, the process eventually converges to a joint optimal solution that satisfies all constraints. The converged intermediate decisions for container start / stop and GPU slicing frequency are then combined and output to form energy-saving action instructions. These instructions consist of two parts: a list of container start / stop instructions and a list of GPU slicing frequency instructions.

Claims

1. A scenario-based AI task scheduling method for business halls based on computing power tiers, characterized in that, include: Based on the video stream access list and algorithm subscription configuration list of the business hall, the scene feature vector is extracted and classified according to the computing power consumption calculation and preset threshold. The computing power demand level label is determined, and the container image pre-pull and graphics processor space partition configuration of the corresponding level are triggered to obtain the pre-ready resource set. Based on the pre-ready resource set and the computing power requirement level label, the predicted latency set of each scheduled task on the central processing unit container path and the graphics processing unit slice path is predicted by the heterogeneous execution latency prediction model library. Tasks with insufficient prediction samples are downgraded and assigned according to the computing power requirement level label to obtain a heterogeneous latency prediction result set. Based on the heterogeneous latency prediction result set, a batch critical path minimization strategy with hierarchical preference penalty and fragment awareness is adopted to perform task-carrier-slice assignment, and the corresponding container is launched through the container orchestration engine to form a heterogeneous running container set. Based on the heterogeneous set of runtime containers, video stream inference is performed, and the measured latency is filtered for outliers and fed back to the heterogeneous execution latency prediction model library to output a standardized business event stream. For the standardized business event flow and real-time power consumption, a limited-length real-time deficit queue and a limited-length energy consumption deficit queue are maintained. The energy-saving action command is obtained by online solution with adaptive trade-off parameters and drift penalty. According to the energy-saving action instructions, the computing power layer performs container start-up and shutdown and adjusts the graphics processor slicing frequency or sleep mode to form the computing power state after energy saving, which serves as the feedback input for the scene feature vector extraction in the next cycle.

2. The scenario-based AI task scheduling method for business halls based on computing power tiers as described in claim 1, characterized in that, The process involves extracting scene feature vectors based on the video stream access list and algorithm subscription configuration list of the business hall, classifying them according to computing power consumption calculations and preset thresholds, determining computing power demand level labels, and triggering the pre-fetching of container images and the configuration of graphics processor space partitions for the corresponding level to obtain a pre-ready resource set, including: The video stream access list and the algorithm subscription configuration list of the business hall are structured and parsed to extract scene feature vectors containing the number of cameras, the resolution of each camera, the frame rate of each camera, and the set of subscription algorithm types. Based on the scene feature vector, each subscription algorithm type in the subscription algorithm type set is mapped to three complexity levels: basic recognition, multimodal analysis, and high-precision composite scene through a preset algorithm complexity mapping table. The resolution of each path, the frame rate of each path, and the complexity level of the corresponding algorithm are multiplied and weighted to obtain the comprehensive computing power requirement value. The comprehensive computing power requirement value is compared with a preset first computing power threshold and a second computing power threshold. Where the first computing power threshold is less than the second computing power threshold, the comprehensive computing power requirement value is determined to be low computing power requirement level when it is less than the first computing power threshold, medium computing power requirement level when it is between the first computing power threshold and the second computing power threshold, and high computing power requirement level when it is greater than the second computing power threshold. The computing power requirement level label is then output. According to the computing power requirement level tag, the level-resource mapping table is queried. The low computing power requirement level triggers the pre-fetching of the CPU container image containing only basic identification inference processes and divides the graphics processor into multiple slice partitions according to the first maximum video memory capacity or puts it into hibernation. The medium computing power requirement level triggers the joint pre-fetching of the CPU container image and the graphics processor slice container image and divides the graphics processor into mixed specification slice partitions according to the second maximum video memory capacity, wherein the second maximum video memory capacity is greater than the first maximum video memory capacity. The high computing power requirement level triggers the pre-fetching of the graphics processor container image containing complete multimodal analysis and high-precision composite scene inference processes and divides the graphics processor into slice partitions according to the third maximum video memory capacity or configures it as a dedicated whole card, wherein the third maximum video memory capacity is greater than the second maximum video memory capacity. The pre-ready resource set is obtained.

3. The scenario-based AI task scheduling method for business halls based on computing power tiers as described in claim 2, characterized in that, The process involves predicting the set of predicted latencies for each scheduled task on the CPU container path and GPU slice path using a heterogeneous execution latency prediction model library, based on the pre-ready resource set and the computing power requirement level label. Tasks with insufficient prediction samples are downgraded and assigned according to the computing power requirement level label, resulting in a heterogeneous latency prediction result set, including: Using the pre-ready resource set and the computing power demand level label as input, for each AI task in the current batch to be scheduled, extract the corresponding algorithm type, input resolution, batch size, target carrier type and current carrier occupancy rate to form the predictor input feature tuple; The predictor input feature tuple is input into the heterogeneous execution latency prediction model library, wherein the heterogeneous execution latency prediction model library is obtained by gradient boosting regression fitting of pre-collected calibration samples, and outputs the first predicted latency of each AI task on the central processing unit container path and the second predicted latency on each available graphics processor slice path. For each AI task, it is determined whether the number of calibration samples corresponding to the predictor input feature tuple in the heterogeneous execution latency prediction model library is less than a preset sample number threshold. If it is less than the preset sample number threshold, a downgrade assignment is triggered. The AI ​​tasks with low computing power requirements are bound to the CPU container path, the AI ​​tasks with medium computing power requirements are bound to the GPU slice path with the middle remaining capacity in the pre-ready resource set, and the AI ​​tasks with high computing power requirements are bound to the GPU slice path with the largest remaining capacity in the pre-ready resource set. The assignment path of the AI ​​task is written back as the latency prediction result of the AI ​​task. If it is not less than the preset sample number threshold, the first prediction latency and the second prediction latency set are retained as the latency prediction result of the AI ​​task. The latency prediction results of all AI tasks are summarized, and the heterogeneous latency prediction result set is output.

4. The scenario-based AI task scheduling method for business halls based on computing power tiers as described in claim 1, characterized in that, Based on the heterogeneous latency prediction result set, a batch critical path minimization strategy with hierarchical preference penalty and fragment awareness is used to perform task-carrier-slice assignment, and the corresponding containers are launched through the container orchestration engine to form a heterogeneous runtime container set, including: Using the heterogeneous latency prediction result set as input, an objective function for assignment decision is constructed. The objective function is the weighted sum of the maximum of the cumulative latency of the CPU path within the batch and the cumulative latency of each GPU slice path, a level preference penalty term, an internal slack term, and an external fragmentation term. The level preference penalty term quantifies the degree to which the assignment path of all AI tasks in the current batch deviates from the computing power requirement level label. The internal slack term quantifies the difference between the allocated computing power of the GPU slice occupied by each AI task and the actual computing power consumed by the corresponding AI task. The external fragmentation term quantifies the amount of space in the GPU that cannot be pieced together to form an effective slice. With minimizing the objective function as the optimization objective, a two-stage solution is performed: In the first stage, when a task arrives, the greedy dispatcher completes the initial task-carrier-slice binding in milliseconds; in the second stage, an asynchronous background thread performs 2-opt local swaps on the binding results of a number of pre-set batches closest to the current optimization time in seconds to calculate the objective function value after the swap. The swap result is submitted only when the objective function value after the swap is lower than the objective function value before the swap, thus obtaining the task-carrier-slice binding decision set. Based on the task-carrier-slice binding decision set, the container orchestration engine issues container start / stop instructions and graphics processor slice activation instructions to the computing layer. For AI tasks bound to the CPU container path in the task-carrier-slice binding decision set, the corresponding CPU virtualization container is started. For AI tasks bound to the CPU slice path in the task-carrier-slice binding decision set, the algorithm container with multi-process service enabled is activated on the specified graphics processor slice, forming the heterogeneous running container set.

5. The scenario-based AI task scheduling method for business halls based on computing power tiers according to claim 4, characterized in that, The process involves, based on the task-carrier-slice binding decision set, issuing container start / stop instructions and graphics processor slice activation instructions to the computing layer via the container orchestration engine. For AI tasks bound to the CPU container path in the task-carrier-slice binding decision set, the corresponding CPU virtualization container is started. For AI tasks bound to the CPU slice path in the task-carrier-slice binding decision set, multi-process service-enabled algorithm containers are activated on the specified graphics processor slice, forming the heterogeneous runtime container set, including: Based on the task-carrier-slice binding decision set, determine whether the target computing node's graphics processor supports multi-instance graphics processor hardware partitioning capability; When the graphics processor of the target computing node supports the hardware partitioning capability of multi-instance graphics processors, the hardware isolation slice is drawn by the multi-instance graphics processor function according to the slice specification specified in the task-carrier-slice binding decision set, and the multi-process service is activated on the hardware isolation slice to perform process reuse in the time dimension, so as to obtain a hardware isolation mode container instance. When the graphics processor of the target computing node does not support the hardware partitioning capability of multi-instance graphics processors, a software slice is generated by the unified computing device architecture flow mechanism according to the slice specification specified in the task-carrier-slice binding decision set, and a multi-process service is activated on the software slice to perform process reuse in the time dimension, thereby obtaining a software slice mode container instance. The hardware isolation mode container instance or the software slicing mode container instance is merged with the CPU virtualization container launched corresponding to the AI ​​task bound to the CPU container path in the task-carrier-slice binding decision set to form the heterogeneous running container set.

6. The scenario-based AI task scheduling method for business halls based on computing power tiers according to claim 1, characterized in that, The process of performing video stream inference based on the heterogeneous runtime container set, filtering out outliers, and feeding back the measured latency to the heterogeneous execution latency prediction model library, outputting a standardized business event stream, includes: Using the heterogeneous set of running containers as input, real-time video stream slices from the perception layer are grouped by timestamp and pushed into the inference task queues corresponding to each container. The inference task queues bound to the central processing unit virtualization container adopt the first batch size, and the inference task queues bound to the graphics processor slice container adopt the second batch size. The second batch size is larger than the first batch size, thus obtaining the inference task queues corresponding to each container. The algorithm inference engine in each container performs target detection, behavior recognition or multimodal fusion inference on the video stream slices in the inference task queue. The measured single inference latency is recorded for each inference, and the latency sample set is summarized to output the frame-by-frame AI analysis results. A sliding window is applied to the time delay sample set, and outlier samples with a median absolute deviation exceeding three times the median absolute deviation are removed from the window. The remaining time delay samples are then fed back to the heterogeneous execution time delay prediction model library for incremental updates of the predictor coefficients. The frame-by-frame AI analysis results are structured and encapsulated by the application layer business gateway. The structured encapsulation includes adding camera identifier, timestamp, event type and confidence field, and outputting the standardized business event stream.

7. The scenario-based AI task scheduling method for business halls based on computing power tiers as described in claim 2, characterized in that, For the standardized business event flow and real-time power consumption, a limited-length real-time deficit queue and a limited-length energy consumption deficit queue are maintained. Through online solution with drift penalty and adaptive trade-off parameters, energy-saving action instructions are obtained, including: Using the standardized business event flow and the real-time power consumption collected by the computing power layer as input, the utilization rate of each container, the frequency of event triggering, the number of active video channels and the instantaneous power consumption are statistically analyzed according to a preset sliding window period to form a real-time load-energy consumption profile. Based on the event processing latency and instantaneous power consumption in the real-time load-energy consumption profile, update the limited-length real-time deficit queue and the limited-length energy consumption deficit queue. The current value of the limited-length real-time deficit queue is the sum of the difference between the previous time slot value and the event processing latency exceeding the service level target threshold, and is constrained by an upper bound. The current value of the limited-length energy consumption deficit queue is the sum of the difference between the previous time slot value and the instantaneous power consumption exceeding the target power consumption upper limit, and is constrained by an upper bound. This yields the updated limited-length real-time deficit queue and the updated limited-length energy consumption deficit queue. Based on the updated time-limited real-time deficit queue and the updated time-limited energy consumption deficit queue, a drift plus penalty target is constructed. The drift plus penalty target is the sum of the time-slot change of the sum of the squares of the updated time-limited real-time deficit queue and the updated time-limited energy consumption deficit queue, multiplied by the adaptive trade-off parameter and the instantaneous power consumption. The adaptive trade-off parameter is adjusted inversely to the current value of the updated time-limited real-time deficit queue. The larger the current value of the updated time-limited real-time deficit queue, the smaller the adaptive trade-off parameter. The smaller the current value of the updated time-limited real-time deficit queue, the larger the adaptive trade-off parameter. Thus, the drift plus penalty target is obtained. With the goal of minimizing the drift plus penalty target as the solution objective, block coordinate descent is used to decouple the decision into container start / stop sub-decision and graphics processor slice frequency or hibernation sub-decision, and the energy-saving action command is output through iterative solution.

8. The scenario-based AI task scheduling method for business halls based on computing power tiers as described in claim 7, characterized in that, The goal is to minimize the drift plus penalty objective. Block coordinate descent is used to decouple the decision into container start / stop sub-decision and GPU slice frequency or hibernation sub-decision. The energy-saving action command is iteratively solved and output, including: Using the drift plus penalty target as input, and fixing the current value of the graphics processor slice frequency or the sleep sub-decision, the container start-stop sub-decision is solved. The decision space of the container start-stop sub-decision includes disabling redundant CPU containers, migrating the container corresponding to the medium computing power demand level task from the graphics processor slice to the CPU container, and keeping only the container corresponding to the basic recognition inference process on standby, thus obtaining the intermediate decision of container start-stop. Using the container start / stop intermediate decision as input, and fixing the container start / stop intermediate decision, the graphics processor slice frequency or sleep sub-decision is solved. The decision space of the graphics processor slice frequency or sleep sub-decision includes reducing the frequency of the graphics processor slice to a preset idle level, merging the tasks on multiple graphics processor slices with the current carrier occupancy rate lower than the preset occupancy rate threshold into a single graphics processor slice, and setting the merged idle graphics processor slice to sleep. Each frequency reduction or sleep action is set with a minimum duration constraint to suppress frequent switching jitter, thus obtaining the slice frequency intermediate decision. The intermediate decisions for container start / stop and slice frequency are iterated alternately. In each iteration, the intermediate decisions for slice frequency are updated based on the current intermediate decisions for container start / stop, and the intermediate decisions for container start / stop are updated based on the current intermediate decisions for slice frequency, until the drift plus penalty target converges. The converged intermediate decisions for container start / stop and slice frequency are then merged and output to form the energy-saving action command.

Images