A preemptive scheduling method for heterogeneous multi-core processors based on multi-agent reinforcement learning
By optimizing preemptive scheduling on a heterogeneous multi-core processor platform using multi-agent reinforcement learning, the problems of heterogeneous resource awareness and multi-agent collaboration are solved, achieving efficient task scheduling and overhead control, and improving system performance.
Patent Information
- Application Number
- CN202610282880.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-03-10
- Publication Date
- 2026-05-26
AI Technical Summary
Existing technologies struggle to achieve efficient real-time task scheduling on heterogeneous multi-core processor platforms, particularly in terms of heterogeneous resource awareness, multi-agent collaboration mechanisms, and reward function design. This results in low scheduling efficiency, poor collaboration, and difficulty in controlling overhead.
A multi-agent reinforcement learning approach is adopted. By establishing a state space, action space, and reward function, the computing power cluster is divided and independent agents are assigned. The preemption decision is optimized using a centralized training and distributed execution framework. The discrete soft actor-critic algorithm is combined for collaborative training to achieve cooperation and cost control among agents.
It improves the task scheduling efficiency of heterogeneous multi-core processor platforms, reduces unnecessary preemption and context switching, and enhances the overall performance of the system.
Smart Images
Figure CN122086566A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of real-time embedded system scheduling technology, specifically relating to a preemptive scheduling method for heterogeneous multi-core processors based on multi-agent reinforcement learning. Background Technology
[0002] With the rapid development of mobile computing, the Internet of Things (IoT), and embedded systems, heterogeneous multi-core processors have become mainstream due to their ability to achieve a good balance between performance and power consumption. These platforms typically integrate high-performance large cores and energy-efficient small cores to form an asymmetric computing environment. However, how to efficiently schedule real-time tasks on such heterogeneous platforms, ensuring that all tasks meet deadlines while minimizing system overhead, has become a key challenge that urgently needs to be addressed in the field of real-time systems.
[0003] Fixed preemption points are an effective real-time scheduling optimization technique. Their core idea is to divide the task execution process into several non-preemptible execution segments, allowing preemption only at predetermined fixed positions. Compared to fully preemptive and non-preemptive scheduling, fixed preemption points can effectively reduce context switching overhead and cache-related overhead while ensuring real-time performance. However, most existing fixed preemption point strategies use static or heuristic methods to select preemption points, lacking adaptability to dynamic system load and heterogeneous resource differences, making it difficult to achieve optimal scheduling performance under varying system utilization levels.
[0004] In recent years, multi-agent deep reinforcement learning has demonstrated powerful learning and generalization capabilities in complex sequential decision-making problems and has been successfully applied in fields such as robot control, game theory, and resource scheduling. Introducing multi-agent deep reinforcement learning into real-time scheduling holds promise for dynamically optimizing preemption decisions by learning from historical scheduling experience. However, existing reinforcement learning-based scheduling research is mostly concentrated on homogeneous multi-core platforms or single-processor environments. Extending it to heterogeneous multi-core scenarios faces the following technical challenges: First, state space design is difficult, requiring simultaneous characterization of the real-time urgency of the task and the computational power differences of heterogeneous cores to support agents in distinguishing the decision benefits on different cores; Second, multi-agent collaboration is challenging, as heterogeneous cores have different sensitivities to preemption behavior, and a single global strategy cannot take into account the characteristics of different cores, requiring multiple agents to collaborate in optimization; Third, reward shaping is complex, and how to guide agents to actively reduce unnecessary preemption and context switching overhead while satisfying hard real-time constraints is a multi-objective optimization problem.
[0005] In summary, existing technologies have not fully solved the preemptive scheduling problem based on reinforcement learning in heterogeneous multi-core environments, especially in terms of heterogeneous resource perception, multi-agent cooperation mechanisms, and reward function design. Summary of the Invention
[0006] The purpose of this invention is to provide a preemptive scheduling method for heterogeneous multi-core processors based on multi-agent reinforcement learning, so as to overcome the shortcomings of existing technologies such as low real-time scheduling efficiency, poor cooperation, and difficulty in controlling overhead in heterogeneous multi-core processors.
[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows:
[0008] A preemptive scheduling method for heterogeneous multi-core processors based on multi-agent reinforcement learning includes the following steps:
[0009] Step 1: Establish a system model, including a real-time task set and a heterogeneous multi-core processor platform. Each task consists of a quadruple. Description, in which The worst-case execution time on the benchmark processor. For the task cycle, This is a relative deadline. This is a preset, fixed number of preemption points. Each processor With nominal speed .
[0010] Step 2: Model the heterogeneous multi-core real-time scheduling problem as a partially observable Markov decision process, and define the state space. Action space and reward function State space Includes the first in the ready queue Dynamic attribute encoding of high-priority tasks and platform-wide heterogeneous feature encoding.
[0011] Step 3: Determine the platform based on the processor's rated speed. Divided into multiple computing clusters Each cluster is assigned an independent scheduling agent, forming a multi-agent system.
[0012] Step 4: Train the heterogeneous multi-core computing power cluster multi-agent system using a centralized training and distributed execution framework. During the training phase, a centralized critic network evaluates the value of the joint actions made by large-core and small-core agents in response to preemptive scheduling, and guides the policy networks of each agent to make collaborative updates based on the feedback of this value, so as to optimize the overall performance of the system in terms of job overdue rate, number of preemptions, and context switching overhead. During the execution phase, each cluster agent independently makes preemptive scheduling decisions based on its local policy network and local observation information of the platform.
[0013] Furthermore, in step 1, for the task Divide its execution process into There are 1 non-preemptible execution block, and the standard computational cost of each block is 1. The fixed preemption point is located at the completion time of each execution block.
[0014] Furthermore, the state space construction method in step 2 is as follows:
[0015] Select the queued member with the earliest absolute deadline from the current ready queue. Each task is designated as a feature task. Construct the following normalized features:
[0016] Urgency index: ,in The absolute deadline for the assignment. The current moment;
[0017] Completion indicators: ,in The remaining execution time of the job;
[0018] Relaxation index: .
[0019] Simultaneously construct the platform's full capabilities:
[0020] Current core computing power level for decision-making: ;
[0021] Weighted sum of global idle computing power: ;
[0022] System instantaneous load factor: ,in This represents the average task cycle.
[0023] Final state vector From all characteristic operations Concatenate, then concatenate global features constitute.
[0024] Furthermore, in step 2, the action space It includes two levels of collaborative decision-making: the first level is preemptive decision-making. Where 0 indicates no preemption and 1 indicates preemption; the second level is when... When the time comes, select a preemption point index from the fixed preemption point set of the currently running task.
[0025] Furthermore, in step 2, the reward function The global reward, shared by all agents, consists of an immediate reward and a segment termination reward:
[0026] Instant rewards Define the time interval between adjacent decision points Inside:
[0027]
[0028] in In order to complete the assignments on time, The number of overdue assignments. In order to seize the number of occurrences, For the number of context switches, These are positive weighting coefficients.
[0029] Segment termination reward Defined as:
[0030]
[0031] in This represents the total number of overdue assignments within this segment. The total number of completed assignments. This is the magnification factor.
[0032] Furthermore, in step 3, the computing power cluster is divided based on a processor speed threshold, with cores having a speed higher than the threshold being classified as large-core clusters. The rest are small-core clusters. Assign intelligent agents to large-core clusters Assigning agents to small-core clusters When a scheduling event occurs on a certain core, the decision is made by the agent corresponding to the cluster to which that core belongs.
[0033] Furthermore, in step 4, a discrete soft actor-critic algorithm is used for preemptive scheduling training, and a joint policy value estimation is introduced to promote agent cooperation: when calculating the target value, a mixed distribution of the policies of the two agents with large cores and small cores is used to estimate the soft state value of the next state in heterogeneous multi-core scheduling.
[0034] The present invention has the following beneficial effects:
[0035] Strong heterogeneous perception: The state space explicitly encodes the core computing power level and the weighted sum of idle computing power, enabling the agent to distinguish the decision benefits on different cores.
[0036] Good synergy: Multi-agents are divided according to computing power clusters, and natural cooperation between large cores and small cores is achieved by sharing global rewards and joint value estimation.
[0037] Advantages in overhead control: The reward function incorporates preemption counts and context switching penalties to guide the agent to minimize system overhead while meeting deadlines. Attached Figure Description
[0038] To clearly illustrate the technical solutions in the embodiments of the present invention and the prior art, the accompanying drawings required for describing the embodiments or the prior art will be briefly described below. Obviously, the following drawings are only some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without creative effort.
[0039] Figure 1 This is a flowchart illustrating the overall process of the method of the present invention.
[0040] Figure 2 This is a schematic diagram of the multi-agent centralized training and distributed execution framework of the present invention. Detailed Implementation
[0041] This invention is a preemptive scheduling method for heterogeneous multi-core processors based on multi-agent reinforcement learning. To make the technical solution, innovations, and beneficial effects of this invention clearer, the implementation methods are described in detail below with reference to specific embodiments. Those skilled in the art can implement the technical solution of this invention based on the content described in this specification without departing from the spirit of this invention.
[0042] A preemptive scheduling method for heterogeneous multi-core processors based on multi-agent reinforcement learning includes the following steps:
[0043] Step 1: Establish a system model, including a real-time task set and a heterogeneous multi-core processor platform. Each task consists of a quadruple. Description, in which The worst-case execution time on the benchmark processor. For the task cycle, This is a relative deadline. This is a preset, fixed number of preemption points. Each processor With nominal speed .
[0044] Step 2: Model the heterogeneous multi-core real-time scheduling problem as a partially observable Markov decision process, and define the state space. Action space and reward function State space Includes the first in the ready queue Dynamic attribute encoding of high-priority tasks and platform-wide heterogeneous feature encoding.
[0045] Step 3: Determine the platform based on the processor's rated speed. Divided into multiple computing clusters Each cluster is assigned an independent scheduling agent, forming a multi-agent system.
[0046] Step 4: Train the heterogeneous multi-core computing power cluster multi-agent system using a centralized training and distributed execution framework. During the training phase, a centralized critic network evaluates the value of the joint actions made by large-core and small-core agents in response to preemptive scheduling, and guides the policy networks of each agent to make collaborative updates based on the feedback of this value, so as to optimize the overall performance of the system in terms of job overdue rate, number of preemptions, and context switching overhead. During the execution phase, each cluster agent independently makes preemptive scheduling decisions based on its local policy network and local observation information of the platform.
[0047] Furthermore, in step 1, for the task Divide its execution process into There are 1 non-preemptible execution block, and the standard computational cost of each block is 1. The fixed preemption point is located at the completion time of each execution block.
[0048] Furthermore, the state space construction method in step 2 is as follows:
[0049] Select the queued member with the earliest absolute deadline from the current ready queue. Each task is designated as a feature task. Construct the following normalized features:
[0050] Urgency index: ,in The absolute deadline for the assignment. The current moment;
[0051] Completion indicators: ,in The remaining execution time of the job;
[0052] Relaxation index: .
[0053] Simultaneously construct the platform's full capabilities:
[0054] Current core computing power level for decision-making: ;
[0055] Weighted sum of global idle computing power: ;
[0056] System instantaneous load factor: ,in This represents the average task cycle.
[0057] Final state vector From all characteristic operations Concatenate, then concatenate global features constitute.
[0058] Furthermore, in step 2, the action space It includes two levels of collaborative decision-making: the first level is preemptive decision-making. Where 0 indicates no preemption and 1 indicates preemption; the second level is when... When the time comes, select a preemption point index from the fixed preemption point set of the currently running task.
[0059] Furthermore, in step 2, the reward function The global reward, shared by all agents, consists of an immediate reward and a segment termination reward:
[0060] Instant rewards Define the time interval between adjacent decision points Inside:
[0061]
[0062] in In order to complete the assignments on time, The number of overdue assignments. In order to seize the number of occurrences, For the number of context switches, These are positive weighting coefficients.
[0063] Segment termination reward Defined as:
[0064]
[0065] in This represents the total number of overdue assignments within this segment. The total number of completed assignments. This is the magnification factor.
[0066] Furthermore, in step 3, the computing power cluster is divided based on a processor speed threshold, with cores having a speed higher than the threshold being classified as large-core clusters. The rest are small-core clusters. Assign intelligent agents to large-core clusters Assigning agents to small-core clusters When a scheduling event occurs on a certain core, the decision is made by the agent corresponding to the cluster to which that core belongs.
[0067] Furthermore, in step 4, a discrete soft actor-critic algorithm is used for preemptive scheduling training, and a joint policy value estimation is introduced to promote agent cooperation: when calculating the target value, a mixed distribution of the policies of the two agents with large cores and small cores is used to estimate the soft state value of the next state in heterogeneous multi-core scheduling.
[0068] Example 1
[0069] This embodiment uses a heterogeneous multi-core platform with four cores as an example, including two large cores (speed 1.5) and two small cores (speed 1.0). Task cycles are evenly distributed between 20 and 50 cycles. Each task has a preset fixed preemption point of no more than 10. A total of 10,000 task sets are generated, divided according to system utilization intervals, with each interval containing 1,000 task sets. Each task set contains... One task, of which This represents the number of processors. During the simulation run, each task continuously generates job instances.
[0070] The multi-agent scheduling strategy proposed in this invention is implemented using PyTorch and trained using GPU acceleration. The training hyperparameters are shown in Table 1.
[0071] Table 1 Hyperparameters
[0072]
[0073] The comparison method uses a naive fixed preemption point strategy, which always selects the nearest indexed fixed preemption point. The evaluation metrics are task set scheduling rate and average context switches. Taking the processor platform as an example, the experimental results are shown in Tables 2 and 3.
[0074] Table 2 Task Set Scheduling Rate
[0075]
[0076] Table 3 Average number of context switches
[0077]
[0078] As shown in Table 2, the two strategies achieve almost identical scheduling capabilities for task sets as system utilization increases. Based on this, Table 3 presents the changes in the average number of context switches for the two strategies—the present invention and the recent index fixed preemption point—as system utilization increases. The present invention demonstrates significant advantages at all system utilization levels. The advantages of the present invention become increasingly apparent as system utilization increases, particularly in the medium (50-70%) and high (70-100%) system utilization ranges, where the strategy significantly reduces the number of context switches.
[0079] In summary, the preemptive scheduling method based on multi-agent reinforcement learning proposed in this invention can significantly reduce unnecessary preemption and context switching while ensuring the scheduling rate of the task set, thereby reducing system overhead and improving the overall performance of the real-time system.
[0080] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A preemptive scheduling method for heterogeneous multi-core processors based on multi-agent reinforcement learning, characterized in that, Includes the following steps: Step 1: Establish a system model, which includes a set of real-time tasks. and heterogeneous multi-core processor platform ; among them, each task By quadruplets Characterization, The worst-case execution time on the benchmark processor. For the task cycle, This is a relative deadline. The number of preemption points is preset to a fixed number; each processor With nominal speed ; Step 2: Model the heterogeneous multi-core real-time scheduling problem as a partially observable Markov decision process, and define the state space. Action space and reward function The state space Includes the first in the ready queue Dynamic attribute encoding of high-priority tasks and platform-wide heterogeneous feature encoding; Step 3: Determine the platform based on the processor's rated speed. Divided into multiple computing clusters Each cluster is assigned an independent scheduling agent, forming a multi-agent system. Step 4: Train the heterogeneous multi-core computing power cluster multi-agent system using a centralized training and distributed execution framework. During the training phase, a centralized critic network evaluates the value of the joint actions made by large-core and small-core agents in response to preemptive scheduling, and guides the policy networks of each agent to make collaborative updates based on the feedback of this value, so as to optimize the overall performance of the system in terms of job overdue rate, number of preemptions, and context switching overhead. During the execution phase, each cluster agent independently makes preemptive scheduling decisions based on its local policy network and local observation information of the platform.
2. The preemptive scheduling method according to claim 1, characterized in that, In step 1, for the task Divide its execution process into There are 1 non-preemptible execution block, and the standard computational cost of each block is 1. The fixed preemption point is located at the completion time of each execution block.
3. The preemptive scheduling method according to claim 1, characterized in that, The state space construction in step 2 includes: Select the queued member with the earliest absolute deadline from the current ready queue. Each task is used as a characteristic task; For each feature job Construct the following normalized features: Urgency Index :,in The absolute deadline for the assignment. The current moment; Completion indicators: ,in The remaining execution time of the job; Relaxation index: ; Construct global characteristics of the platform: Current core computing power level: ; Weighted sum of global idle computing power: ; System instantaneous load factor: ,in This represents the average task cycle. Final state vector From all characteristic operations Concatenate, then concatenate global features constitute.
4. The preemptive scheduling method according to claim 1, characterized in that, The action space It includes two levels of collaborative decision-making: the first level is preemptive decision-making. Where 0 indicates no preemption and 1 indicates preemption; the second level is when... When the time comes, select a preemption point index from the fixed preemption point set of the currently running task.
5. The preemptive scheduling method according to claim 1, characterized in that, The reward function The global reward, shared by all agents, consists of an immediate reward and a segment termination reward: the immediate reward... Define the time interval between adjacent decision points Inside: ， in In order to complete the assignments on time, The number of overdue assignments. In order to seize the number of occurrences, For the number of context switches, These are positive weighting coefficients; The segment terminates the reward Defined as: ， in This represents the total number of overdue assignments within this segment. The total number of completed assignments. This is the magnification factor.
6. The preemptive scheduling method according to claim 1, characterized in that, In step 3, the computing power cluster is divided based on a processor speed threshold, with cores having a speed higher than the threshold being classified as large-core clusters. The rest are small-core clusters. Assign agents to large-core clusters Assigning agents to small-core clusters ; When a scheduling event occurs on a certain core, the decision is made by the agent corresponding to the cluster to which that core belongs.
7. The preemptive scheduling method according to claim 1, characterized in that, In step 4, the discrete soft actor-critic algorithm is used for preemptive scheduling training, and joint policy value estimation is introduced to promote agent cooperation: when calculating the target value, the mixed distribution of the policies of the two agents with large cores and small cores is used to estimate the soft state value of the next state in heterogeneous multi-core scheduling.