Multi-granularity cooperative operator scheduling method for dynamic reasoning task

Abstract

The invention relates to a multi-granularity cooperative operator scheduling method for a dynamic reasoning task, and aims to solve the problems of low GPU resource utilization rate and high reasoning delay of a multi-outlet model in a high-concurrency dynamic scene. The invention provides a multi-granularity collaborative framework of static fine granularity optimization and dynamic coarse granularity scheduling. In a micro level, an operator sequence and stream distribution in a model section are optimized through a time sequence constraint and resource awareness algorithm, and the emission overhead during operation is reduced by utilizing a CUDA Graph technology; in the macroscopic level, a dynamic scheduler based on multi-agent reinforcement learning (MARL) is constructed, and task dequeuing and inter-segment parallel combination are managed by using collaborative double agents. According to the method, the system load and resource competition can be adaptively balanced according to the real-time flow fluctuation and the task progress, the system throughput is remarkably improved, the reasoning delay is reduced, and efficient utilization of GPU resources is achieved.

Description

Technical Field

[0001] This invention relates to the field of computer architecture reasoning acceleration, and more specifically, to a multi-granularity cooperative operator scheduling method based on static compiler optimization and dynamic multi-agent reinforcement learning for dynamic reasoning tasks of multi-exit models. Background Technology

[0002] With the rapid development of deep learning technology, model architectures are becoming increasingly complex. To accelerate model inference, existing deep learning frameworks (such as PyTorch and TensorFlow) primarily focus on optimizing intra-operator parallelism. However, this strategy faces severe challenges under current hardware development trends: on the one hand, GPU hardware performance is advancing rapidly, for example, the peak performance of single-precision floating-point operations on the NVIDIA Tesla A100 has reached 19.5 TFLOPs / s; on the other hand, modern model designs tend to use multi-branch, small operator structures, and the number of operators is surging, leading to a sharp decrease in the computational cost of a single operator. For example, it has decreased from 2330 MFLOPs / kernel in VGG to 82 MFLOPs / kernel in NASNet. This contradiction between "powerful hardware" and "small operators" means that intra-operator optimization alone is insufficient to cover the overhead of operator startup and data transfer, resulting in a serious waste of the streaming multiprocessor (SM) resources within the GPU.

[0003] To address the aforementioned bottlenecks, recent research has shifted towards exploring inter-operator parallelism. Ma et al. proposed a fine-grained operator scheduling based on the Wavefront algorithm and enabled operator fusion on GPU devices. Zhang et al. further coordinated the scheduling of control flow and data flow operators at the compiler level based on Rammer. Regarding search-based systematic scheduling, Ding et al. innovatively used dynamic programming (DP) to divide convolutional neural networks (CNNs) into multiple stages, systematically searching for the optimal strategy for merging or concurrently executing operators in each stage to accelerate inference. Ping et al. extended latency-aware inter-operator scheduling based on IOS. Kwon et al. utilized a bipartite graph algorithm to fully schedule operators onto CUDA streams. The above methods (especially DP and bipartite graph matching) typically require high search or transformation overhead. Opara focuses on solving the runtime overhead problem of operator parallelism, proposing an interference-aware scheduling strategy that achieves efficient inter-operator parallelism through a lightweight stream allocation algorithm, optimized operator startup order, and the use of CUDA Graph to eliminate runtime overhead.

[0004] However, in practical applications, especially in inference scenarios for multi-exit models, tasks arrive dynamically and asynchronously from a large number of terminals, with varying computational loads and real-time requirements. Furthermore, the inference progress of each task within the model also differs. Existing parallel scheduling schemes between operators have significant limitations. Traditional "offline compilation" or "static scheduling" schemes (such as compiling the entire model into a static graph) become suboptimal or even ineffective in the face of this dynamic nature because they cannot predict the task combinations and progress at runtime. On the other hand, using online dynamic scheduling only during inference incurs huge runtime overhead, and frequent scheduling decisions can actually hinder inference performance. Moreover, due to the non-preemptive nature of the CUDA core, an unreasonable operator startup order can lead to GPU blocking and performance interference between operators, potentially even significantly increasing inference latency.

[0005] Currently, no work has solved the operator scheduling problem in multi-exit models under dynamic multi-task scenarios. How to achieve efficient parallelism and resource load balancing of dynamic task flows while reducing runtime scheduling overhead is a key technical challenge for improving the throughput and reducing latency of high-performance computing systems. Summary of the Invention

[0006] To address the issues of low GPU resource utilization, severe task backlog, and high inference latency caused by asynchronous task arrival and large differences in inference progress in multi-exit models under high-concurrency dynamic inference scenarios, this invention proposes a multi-granularity cooperative operator scheduling method for dynamic inference tasks. This method aims to decouple static optimization and dynamic scheduling through a layered architecture, achieving the lowest average latency and improving GPU resource utilization.

[0007] First, the research focuses on high-concurrency inference tasks using multi-exit models on GPU platforms. For this scenario, the inherent relationships between model characteristics, hardware resources, and task flow dynamics are analyzed, leading to the following technical conclusions:

[0008] Conclusion 1: Static optimization is suitable for the internal structure of a model segment. Although the overall task flow is dynamic, the computational graph structure of the sub-models (i.e., model segments) between every two exit points in a multi-exit model is fixed. Simple parallelization can lead to poor performance due to performance interference, but if a timing- and resource-aware execution unit can be pre-generated for each model segment, the micro-scheduling overhead at runtime can be eliminated.

[0009] Conclusion 2: Inter-segment scheduling requires dynamic awareness. In asynchronous multi-task systems, multiple tasks may simultaneously be at different stages of the model. Static scheduling cannot handle this macroscopic randomness. Solving the problems of task queue backlog and inter-segment resource contention requires an intelligent decision-making mechanism capable of real-time awareness of queue status, inference progress, and GPU load. Multi-agent reinforcement learning (MARL) is an effective paradigm for solving such multi-objective dynamic decision-making problems.

[0010] Based on the above analysis, this invention proposes a multi-granularity collaborative operator scheduling framework. This framework decouples complex problems into two levels: "static fine-grained optimization (intra-segment)" and "dynamic coarse-grained scheduling (inter-segment)". At the bottom layer, model segments are compiled into efficient CUDA graphs through time-constraint and resource-aware algorithms; at the top layer, a collaborative dual-agent system is used to dynamically combine these CUDA graphs according to real-time states and manage task queues.

[0011] To achieve the above objectives, the specific technical solution adopted is as follows:

[0012] In a first aspect, the present invention provides a static optimization method for model segments based on time constraints and resource awareness, the method comprising:

[0013] The multi-exit model is segmented, and the network layer between every two exit points is defined as a "segment". For each segment, a resource-aware analyzer is first used to analyze and quantify the multi-dimensional resource attributes of each operator (such as SM utilization, memory bandwidth requirements, and operator execution time).

[0014] Next, a time-constrained scheduler is designed, employing a heuristic algorithm to search for the optimal operator startup order. This algorithm introduces a composite objective function, incorporating critical path time constraints. Resource conflict cost function .in, Ensure that scheduling prioritizes shortening the critical path that has the greatest impact on overall latency. This is used to accurately quantify the degree of competition for multidimensional resources when operators are parallelized, guiding the search to proactively avoid combinations that lead to severe resource congestion (such as avoiding the concurrent execution of two memory-intensive operators), and promoting the overlapping execution of computationally intensive and memory-intensive operators. Simultaneously, a CUDA Stream allocator is designed, using a greedy algorithm to allocate CUDA Streams to operators.

[0015] Finally, the optimal CUDA Stream allocation scheme and operator startup order are captured into a single, reusable CUDA Graph using the CUDA GraphCapture mechanism. This Graph serves as the atomic execution unit for upper-level scheduling, significantly reducing the runtime emission overhead of intra-segment operators and ensuring minimal latency for intra-segment execution.

[0016] Secondly, this invention provides a dynamic scheduling method for inter-segment inference tasks based on multi-agent reinforcement learning (MARL), the method comprising:

[0017] Construct a MARL environment capable of perceiving macroscopic states, state space Includes: Task queue backlog vector (Perception task waiting time), in-transit inference vector (Aware of the current real-time inference progress of all tasks) and global resource status. (Perceive GPU real-time utilization).

[0018] Design two collaborative agents:

[0019] Agent I: Responsible for task dequeuing and resource management. Its action space controls the dequeueing frequency of each task queue. This is designed to prevent blindly accepting tasks under high load and to balance task backlog and system load.

[0020] Agent II (Agent B): Responsible for inter-segment parallel scheduling. Its action space maps model segments to different execution intervals. This agent learns how to dynamically combine optimized CUDA graphs (i.e., execution units generated by the first aspect), such as executing "model segment 1" of task A concurrently with "model segment 3" of task B, to maximize system parallelism.

[0021] Define a unified composite reward function This approach comprehensively considers task throughput, average latency, GPU resource utilization, and queue backlog variance. By maximizing this reward function, two agents are trained to work collaboratively, achieving multi-objective optimization.

[0022] Beneficial effects

[0023] Compared with the prior art, the present invention has the following advantages:

[0024] 1. Extremely low runtime overhead: Through the static optimization of the first aspect, the complex fine-grained operator scheduling is solidified into CUDA Graph, compressing the microsecond-level overhead (Kernel Launch Overhead) originally caused by the CPU issuing operators one by one into the overhead of a single Graph Launch, reducing the runtime scheduling and dependency checks of individual operators, so that dynamic scheduling only needs to handle the coarse-grained "model segment".

[0025] 2. High GPU resource utilization: The multi-granularity collaborative strategy achieves the overlap of heterogeneous operators (computation / memory intensive) through resource conflict awareness at the micro level; at the macro level, it enables the GPU to process tasks at different stages at the same time through inter-segment parallelism of Agent II, filling pipeline voids.

[0026] 3. Significantly reduce inference latency and task backlog: Agent I's proactive queue management effectively prevents task backlog caused by system overload, and combined with optimal static scheduling within segments, it minimizes end-to-end latency.

[0027] 4. Strong adaptability to dynamic scenarios: Unlike traditional static compilation schemes, the MARL framework of this invention can dynamically adjust its strategy according to real-time traffic fluctuations and task progress, and can adaptively handle complex task flows that arrive asynchronously. Attached Figure Description

[0028] Figure 1 To enable overlapping execution of computationally intensive and memory-intensive operators.

[0029] Figure 2 A static collaboration flowchart for generating the CUDA Graph.

[0030] Figure 3 A framework diagram for multi-agent reinforcement learning collaboration. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this invention clearer, the implementation of this invention will be described in detail below with reference to the accompanying drawings and examples.

[0032] This invention designs a multi-granularity collaborative operator scheduling method for dynamic inference tasks in multi-exit models, addressing the performance bottleneck of multi-exit models in asynchronous concurrent scenarios. Traditional deep learning compilers are efficient in processing static graphs but lack dynamic adaptability, while pure dynamic schedulers have excessive overhead. This invention creatively proposes a multi-granularity collaborative approach of "static micro-optimization + dynamic macro-scheduling". The design mainly includes three steps: intra-segment optimization based on time constraints and resource awareness, inter-segment scheduling based on multi-agent reinforcement learning, and system integration.

[0033] Step 1: Design an intra-segment operator scheduling scheme based on time constraints and resource awareness.

[0034] The goal of this step is to solve the static operator scheduling problem within a "segment". A segment is defined as a sub-model DAG graph between two exit points in a multi-exit model.

[0035] Model segmentation and resource quantification: The multi-exit model is divided into... Each model segment Profiling is performed on each segment to obtain each operator within it. Multidimensional resource demand vector This includes, but is not limited to, SM utilization, video memory bandwidth usage, and single operator execution time.

[0036] Intra-segment scheduling algorithm analysis: Modeling a segment as a computational graph The goal of optimization is to find an optimal parallel scheduling plan. This causes the inference delay of this model segment. Minimize. Total latency is broken down into parallel execution time. and synchronization overhead Scheduling plan Further decoupling into a "CUDA Stream allocation scheme" (Primarily aimed at minimizing synchronization overhead. This scheme determines the mapping relationship of operators on CUDA Streams. By aggregating data-dependent operators to the same Stream, unnecessary inter-stream synchronization can be eliminated; simultaneously, dependency-free operators are distributed across different Streams to enable hardware-level concurrent execution mechanisms.) and "Operator startup order". (Traditional deep learning frameworks like PyTorch only execute sequentially based on data dependencies, such as executing computationally intensive matrix multiplication operators like MatMul and memory-intensive activation operators like ReLU in turn, resulting in alternating periods of idle computation units and GPU memory bandwidth. The startup sequence of this invention...) The computationally intensive operator MatMul and the memory-intensive operator ReLU are interleaved in their issue timing. This sequence, combined with a multi-stream concurrency mechanism, allows MatMul to fully utilize the Tensor Cores for computation while ReLU can utilize idle memory bandwidth in parallel for data throughput, achieving coordinated utilization of peak hardware resource performance. The optimization objective formula is:

[0037] Heuristic scheduling algorithm design: to solve for the optimal order This invention designs a heuristic algorithm, the core of which lies in introducing a critical path analysis function. and resource cost conflict function .

[0038] This is used to quantify the timing constraints and critical path delays under the current scheduling scheme. Assume the set of operators within the model segment is... Operator The execution time is First, based on the dependency graph between operators and the profiling of the scheduling order... Calculate the earliest start time (EST) and earliest end time (EFT) for each operator:

[0039]

[0040]

[0041] in, express The set of predecessor operators; In order Next, operator The point in time when the required hardware resources become idle. Based on this, Defined as the maximum end time of the entire computation graph:

[0042]

[0043] for To accelerate search convergence, relaxation (Slack) is introduced. Operators are defined. Latest start time In order to not postpone Given the latest allowed startup time under the premise of [condition], the operator's relaxation is:

[0044]

[0045] when At that time, the operator is on the critical path. The heuristic search algorithm will prioritize adjusting... The operator in The position in the middle, for direct optimization .on the other hand, Used to quantify the penalty for resource contention when two operators are executed simultaneously. Cosine similarity is used to measure the competition for resource demands between two operators. Representing operators respectively (resource demand vector)

[0046]

[0047] in These represent different resource dimensions. Next, a composite objective function is constructed. :

[0048] in ,at the same time and Both of these were standardized using Z-score.

[0049] objective function Minimize critical path latency ( ) and minimizing hardware resource conflicts ( A comprehensive approach was taken, and the goal was to minimize... It can guide heuristic algorithms to search for an optimal operator startup order that satisfies timing constraints and maximizes the complementary parallelism of heterogeneous resources (computation / memory access). This serves as the solidification order for the final generated CUDA Graph.

[0050] Weighting coefficient and The determination adopts an "automatic tuning strategy based on offline analysis". Specifically, during the compilation phase before model deployment, the system constructs a hyperparameter search space (setting...). ,and For each candidate weight combination, the system generates a corresponding temporary CUDA Graph and performs a trial run on the target hardware. Ultimately, the system selects the combination that minimizes the measured inference latency. and The value is used as the scheduling parameter for the model segment in the current hardware environment. This method can adaptively balance the trade-off between "critical path priority" and "resource conflict avoidance" to ensure that the generated static graph achieves optimal performance on specific hardware.

[0051] Greedy stream allocation algorithm design: Operators without dependencies are allocated to multiple CUDA Streams to maximize operator parallelism, while operators with data dependencies are allocated to the same CUDA Stream, eliminating the need for CUDA Stream synchronization for dependent operators. The final stream allocation scheme is obtained. .

[0052] Generate execution units (CUDA Graph): Using the algorithm described above, the system will identify operator pairs suitable for parallelization (such as...). Figure 1As shown, the memory-intensive ReLU operators and computationally-intensive MatMul operators that were originally independent on a single CUDA Stream are distributed across multiple CUDA Streams for overlapping execution. The priority of critical operator emission is adjusted based on the critical path, and a stream allocation plan is generated. Finally, as shown... Figure 2 As shown in the static collaboration flowchart, the "CUDA Stream allocation scheme" is implemented using CUDA Graph Capture technology. And "operator startup order" ,form The scheduling scheme is captured and summarized into a single, reusable CUDA Graph. ,Right now This Graph will serve as the atomic action of the agent in step 2, compressing the microsecond-level overhead (Kernel Launch Overhead) originally caused by the CPU issuing operators one by one into a single Graph Launch overhead, greatly reducing the scheduling burden of the CPU at runtime.

[0053] Step 2: Construct an inter-segment dynamic scheduling framework based on multi-agent reinforcement learning

[0054] After completing the intra-segment static optimization, this step addresses the dynamic scheduling problem in high-concurrency task flows. In real-world asynchronous multi-task inference scenarios, multiple tasks may be in different stages (i.e., different segments) of the same model at the same time. This section will construct an inter-segment operator scheduling framework that is aware of inference progress. Figure 3 As shown, the framework uses two agents to collaborate and perform dynamic macro-scheduling based on the real-time progress of the inference task, taking into account the backlog of the task queue and hardware resources.

[0055] Environmental modeling:

[0056] intelligent agents at all times Observed state It forms the basis for decision-making:

[0057] , represents the queue backlog vector, where yes Time Queue The total waiting time corresponds to the backlog of tasks.

[0058] , represents the in-transit inference vector, where It is currently in progress Execution The number of tasks corresponds to the perception of reasoning progress.

[0059] This represents the global resource status, which is the resource monitoring agent. The input, which includes occupancy rate Memory bandwidth .

[0060] Motion Design:

[0061] Actions of two agents Combinatorial definition:

[0062]

[0063] ,express The outlining action, Defined as a queue The outgoing frequency or interval. By adjusting this A frequency to respond Queue backlog .

[0064] ,express The inter-segment scheduling action will Defined as a continuous value. The interval is divided into The action space is a contiguous interval. Agent B's action aims to assign an execution time window index (or parallel group ID) to each model segment to be scheduled. Although the action space is continuous, through discretization mapping, the system treats model segments within the same time window as a parallel batch, i.e., all corresponding... Model segments whose values ​​fall within the same interval will be grouped together as a single, parallel execution phase. If Agent B maps too many segments of the same type to the same window (e.g., two resource-intensive segments), this will cause issues. and Mapping to the same interval will lead to severe resource contention and increased latency. This negative consequence will be reflected in the reward function. Feedback is given to the policy network.

[0065] Reward Mechanism:

[0066] reward function Used to guide the learning of intelligent agents, balancing three objectives: high throughput, low latency, and high efficiency. Resource utilization rate. Expressed as an exponential form normalized by the Z-score:

[0067]

[0068] : Task throughput, the total number of tasks completed at time t.

[0069] Average delay: The end-to-end average delay of a task completed at time t.

[0070] GPU resource utilization Average utilization rate.

[0071] Queue backlog penalty. Total queue length. The variance. High variance indicates that some queues are experiencing severe backlogs of inference tasks and should be penalized.

[0072] The weights of each item.

[0073] The two agents share the reward function, and Agent A uses the penalty term. Learning to smooth traffic, Agent B, through... and Learn the optimal parallel strategy.

[0074] Strategy Update:

[0075] To address the non-stationarity issue and achieve efficient collaboration in multi-agent environments, this invention employs a centralized training, decentralized execution (CTDE) architecture. The MAPPO (Multi-Agent PPO) algorithm is used for network updates.

[0076] Critic Value Network (Centralized): Set up a global Critic network. It receives the complete global state. As input, it is responsible for evaluating the value of the current global state, represented as .

[0077] Actor Policy Network (Distributed): Agent A and Agent B each have their own independent Actor policy network. and ,in Represents local observations of Agent A. Representing Agent B's local observations, based on their respective local observations (in this invention, local observations are consistent with the global state). ,Right now Output actions.

[0078] The agent's parameter updates aim to maximize the cumulative reward. The Clip objective function of the PPO algorithm is used to ensure training stability. For each agent... The update formula for its Actor policy network is as follows, i.e., the loss function. for:

[0079]

[0080] These are the weight parameters of the Actor network;

[0081] The expected value represents the average performance on the current batch of sampled data;

[0082] It is the probability ratio of the old and new strategies. Indicates the state Output action after updating network parameters The probability, This indicates the network's state before the parameter update. Down Output Action The probability (or probability) measures the size of the step size in this parameter update. If it is greater than 1, it means the new policy is more inclined to this action; if it is less than 1, the opposite is true.

[0083] This is a pruning hyperparameter (usually set to 0.2) to prevent excessively large policy update magnitudes;

[0084] It is the Generalized Advantage Estimation (GAE), which represents "the best action to take in the current state". "How much better than the average level?" If A value of 0 indicates a good scheduling decision (e.g., successfully reducing latency), and the probability of this action should be increased. If If this is the case, it indicates that this is a poor decision (such as causing memory overflow), and the probability of this action should be reduced. Calculated from a centralized Critic value network:

[0085]

[0086] The forward time step represents the time from the current moment. Starting point, the number of steps forward;

[0087] Discount factor, set to 0.99;

[0088] Smoothing factor, set to 0.95;

[0089] Attenuation coefficient;

[0090] (Reward): In the future Instant rewards earned with each step;

[0091] (Current Value): Critic network pair Estimation of the value of a state at any given moment;

[0092] (Future Value): The potential value brought about by entering the next state after actually executing one step;

[0093] In this formula, the term It represents The temporal difference error at time step is the difference between the 'real reward' obtained by the system after performing an action and the 'expected reward' of the Critic network.

[0094] The update of the Critic value network is achieved by minimizing the mean squared error (MSE) between the predicted value and the actual return, and its loss function is... for:

[0095]

[0096] The weight parameters of the Critic network;

[0097] Expected value: representing the average performance on the current batch of sampled data;

[0098] : Critic's predicted score;

[0099] : Indicates the actual reward target, which is the calculated actual cumulative reward.

[0100] The goal is to make the "predicted score" get closer and closer to the "true score".

[0101] The synergistic effect of the loss functions of the Actor network and the Critic network:

[0102] The aforementioned Actor loss function The aim is "policy improvement," which seeks the optimal scheduling action by maximizing cumulative rewards; and the Critic loss function defined here... The aim is to “calibrate value” by minimizing the mean squared error (MSE) to force the value network to accurately fit the system’s true return curve.

[0103] The two form a closed loop through generalized advantage estimation (GAE): the more accurate the Critic ( The smaller the value, the greater the calculated advantage. The more credible the strategy, the more effectively it guides the Actor to update its strategy. (Convergence), ultimately achieving a steady improvement in the operator scheduling strategy.

[0104] Step 3: System Integration and Logic Verification

[0105] Closed-loop control process: During system operation, GPU status and queue information are collected in real time.

[0106] S1: Perception: Obtain the current situation .

[0107] S2: Decision: Agent A decides how many tasks to release from the new task queue and the intermediate queue; Agent B decides how to combine the model segments (i.e., the CUDA Graphs generated in step 1) corresponding to these released tasks.

[0108] S3: Execution: The CPU emits selected CUDA Graphs to the GPU in parallel via the CUDA API.

[0109] S4: Feedback: Monitor throughput, latency, and utilization after execution, and calculate rewards. Update the agent policy network.

[0110] Through the above steps, this invention achieves efficient management of asynchronous inference task flows for multi-exit models, solving the problem that traditional static scheduling cannot cope with dynamic loads.

Claims

1. A multi-granularity cooperative operator scheduling method for dynamic reasoning tasks, characterized in that, The method includes the following steps: Step 1: Construct an intra-segment static optimization scheme based on time constraints and resource awareness. The multi-exit model is segmented, with each network layer between two exit points defined as a model segment. For each model segment, the multidimensional resource attributes of the operators are quantified. The operator startup order is searched using a heuristic algorithm. and CUDA Stream allocation scheme and the scheduling scheme Capture it as a reusable CUDA Graph; Step 2: Construct an inter-segment dynamic scheduling framework based on multi-agent reinforcement learning Constructing an environment that perceives macroscopic states, state space This includes the task queue backlog vector, in-transit inference vector, and global resource state; designing two collaborative agents, Agent A and Agent B, to manage task dequeueing and inter-segment parallel scheduling, respectively; and defining the reward function. Policy updates are performed using a multi-agent algorithm; Step 3: Execute closed-loop control The system collects status data in real time during operation. Use the trained agent to output actions It emits a selected CUDA Graph to the GPU via the CUDA API and continuously optimizes based on feedback.

2. The method according to claim 1, characterized in that, In step 1, the heuristic algorithm introduces a composite objective function. The formula is as follows: in, This is a critical path analysis function used to quantify the critical path delay under the current scheduling scheme, defined as the maximum end time of the entire computation graph. ; The resource conflict cost function is calculated based on the cosine similarity of resource demands between operators, and is used to quantify the degree of resource competition when operators are parallel. and The hyperparameters for balancing the weights.

3. The method according to claim 1, characterized in that, In step 1, the CUDA Graph generation process includes: using the CUDA Graph Capture mechanism to calculate the optimal flow allocation scheme. and operator startup order Solidified into a single graph structure At runtime, a single Graph Launch replaces the Kernel Launch for each operator, reducing CPU launch overhead.

4. The method according to claim 1, characterized in that, In step 2, the actions of the two agents are defined as follows: Actions of Agent A Controlling the dequeue frequency of each task queue to balance task backlog and system load; the actions of Agent B. By mapping model segments to different execution time windows, the system treats model segments within the same time window as a parallel batch and executes them in an overlapping manner.

5. The method according to claim 1, characterized in that, In step 2, the composite reward function The exponential form using Z-score normalization is defined by the following formula: in, For normalized task throughput, The normalized average delay, For normalized GPU resource utilization, For the normalized queue backlog variance, These are the weighting coefficients for each item.

6. The method according to claim 1, characterized in that, In step 2, the policy update adopts a centralized training, distributed execution (CTDE) architecture, specifically including: setting up a global Critic network. Receive global state And calculate the generalized dominance estimate Agent A and Agent B each have independent Actor policy networks, which output actions based on local observations. Using the MAPPO algorithm, the Actor network parameters are updated through the Clip objective function of PPO, and the Critic network parameters are updated through the mean squared error loss.

7. A multi-granularity cooperative operator scheduling system for dynamic reasoning tasks, characterized in that, The system includes: The static compilation module is used to segment and analyze resources in the multi-exit model, generate a static scheduling scheme using time-constraint and resource-aware algorithms, and encapsulate it as a CUDA Graph execution unit. The environmental awareness module is used to monitor the backlog of tasks in the queue in real time. In-transit reasoning task and GPU global resource status ; The collaborative scheduling module includes a task management agent (Agent A) and an inter-segment parallel agent (Agent B), which are used to dynamically output dequeue instructions and parallel combination strategies according to the environmental state. The execution control module receives instructions from the cooperative scheduling module, calls the CUDA API to launch the corresponding CUDA Graph to the GPU for execution, and returns the execution results.

Images