Multitask scheduling method and device for real-time operating system based on reinforcement learning

Abstract

The invention discloses a real-time operating system multi-task scheduling method and device based on reinforcement learning, and relates to the technical field of operating systems. The method comprises the steps of collecting state information of each task in an operating system kernel in real time; wherein the task state information comprises task attributes and required system resource information; converting the task state information into a state vector, and inputting the state vector into a reinforcement learning model for reasoning, so as to obtain a scheduling decision based on a State-Action table; and updating the State-Action table according to the reward obtained by executing the scheduling decision. Through dynamic scheduling and task priority weighting, high-priority tasks (such as obstacle avoidance tasks) are ensured to be executed preferentially, and real-time response of the system is ensured, so that an embedded system is supported.

Description

Technical Field

[0001] This application relates to the field of operating system technology, specifically to a method and apparatus for multi-task scheduling in a real-time operating system based on reinforcement learning. Background Technology

[0002] In real-time operating systems, task scheduling algorithms are responsible for determining when and how to schedule tasks. While existing real-time scheduling algorithms (such as static priority scheduling and Completely Fair Scheduler (CFS)) can effectively manage task scheduling, they suffer from scheduling latency, resource waste, and low task execution efficiency when facing dynamic loads and systems with high real-time requirements. With the widespread adoption of Linux operating systems, especially in scenarios requiring high real-time performance (such as drone swarms and robot collaboration), traditional scheduling algorithms struggle to meet the demands of complex scenarios involving multiple tasks, dynamic loads, and resource constraints. For example, traditional scheduling methods typically rely on static priorities and fixed scheduling policies, failing to dynamically adjust based on system load and task characteristics, resulting in poor system responsiveness and real-time performance. Furthermore, in multi-tasking environments, existing scheduling algorithms often ignore resource contention between tasks, leading to task execution latency and system performance degradation. Additionally, existing scheduling algorithms are primarily designed for data center or cloud computing environments, failing to meet the needs of resource-constrained embedded and drone applications with high real-time performance requirements. Summary of the Invention

[0003] This application aims to disclose a method and apparatus for multi-task scheduling in a real-time operating system based on reinforcement learning. It can realize dynamic task scheduling and adaptive resource allocation in operating systems with high real-time requirements. Through the training and inference of reinforcement learning models, it can efficiently manage resources such as task priority, CPU affinity, and memory allocation, thereby achieving stable operation under multi-task and high real-time load.

[0004] To achieve the above objectives, this application adopts the following technical solution: Firstly, a multi-task scheduling method for a real-time operating system based on reinforcement learning is disclosed, including: Real-time acquisition of status information for each task in the operating system kernel; the task status information includes task attributes and required system resource information; The task state information is transformed into a state vector and input into the reinforcement learning model for inference, so as to obtain scheduling decisions based on the State-Action table; The State-Action table is updated based on the reward obtained from executing the scheduling decision; The reinforcement learning model uses a State-Action table to store the expected reward value for each possible action in different tasks, and selects the action with the highest expected reward value from the State-Action table to execute based on the current task. After task execution, the reinforcement learning model calculates the task's execution reward to update the State-Action table. The execution reward calculation includes: Calculate the reward value based on the reward function; The reward value is adjusted based on the task status information.

[0005] In one example, the reward function is: reward = θ1 × fps_gain + θ2 × (1 - e_consumed) -θ3 × latency_penalty Where reward is the reward value, fps_gain is the frame rate gain value, e_consumed is the energy consumption value, latency_penalty is the latency penalty value, and θ1, θ2, and θ3 are the weights.

[0006] In one example, the task attributes include task priority, and the system resource information includes one or more of CPU usage, GPU usage, memory usage, and context switching count; the reward value is adjusted based on the task priority and one or more of the system resource information.

[0007] In one example, adjusting the reward value based on task priority includes increasing the reward value by multiplying the task priority's impact value by its weight; adjusting the reward value based on system resource information includes decreasing the reward value by multiplying the resource usage's impact value by its weight.

[0008] In one example, the actions stored in the State-Action table include adjusting task priority, setting CPU affinity, adjusting scheduling cycle, modifying task memory allocation, and not making any adjustments.

[0009] In one example, the State-Action values ​​in the State-Action table are updated using the following method: State-Action[state][action] = (1 - α) × State-Action[state][action]+ α × (reward + γ × max_value_next) Where State-Action[state][action] is the State-Action value corresponding to the task state and action, α is the learning rate, γ is the discount factor, and max_value_next represents the maximum future reward value.

[0010] Secondly, a real-time operating system multi-task scheduling device based on reinforcement learning is disclosed, comprising: The information acquisition module collects the status information of each task in the operating system kernel in real time; the task status information includes task attributes and required system resource information. The decision-making module transforms task state information into state vectors and inputs them into the reinforcement learning model for reasoning, so as to obtain scheduling decisions based on the State-Action table. The update module updates the State-Action table based on the rewards obtained from executing the scheduling decision; The reinforcement learning model uses a State-Action table to store the expected reward value for each possible action in different tasks, and selects the action with the highest expected reward value from the State-Action table to execute based on the current task. After task execution, the reinforcement learning model calculates the task's execution reward to update the State-Action table. The execution reward calculation includes: Calculate the reward value based on the reward function; The reward value is adjusted based on the task status information.

[0011] The beneficial effects of this application are as follows: This application ensures that high-priority tasks (such as obstacle avoidance tasks) are executed first by using dynamic scheduling and task priority weighting, thereby guaranteeing the real-time response of the system and supporting embedded systems.

[0012] This application intelligently allocates resources based on the resource requirements of the task and the system load, avoiding excessive resource consumption and task delays. The intelligent scheduling algorithm ensures the stability of the system in complex environments and avoids task failures or crashes caused by resource requisition.

[0013] Other features and advantages of this application will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0014] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1This is a flowchart illustrating a real-time operating system multi-task scheduling method based on reinforcement learning according to an embodiment of this application. Detailed Implementation

[0015] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0016] According to an embodiment of this application, a real-time multi-task scheduling method for operating systems based on reinforcement learning is disclosed. This method is applied to scenarios with high real-time requirements, such as task scheduling and system resource allocation in Linux operating systems, including drone swarms, multi-robot systems, and IoT devices. It aims to improve the real-time response capability, throughput, and energy efficiency of the operating system through adaptive scheduling strategies and resource optimization.

[0017] Figure 1 This is a schematic flowchart of a reinforcement learning-based real-time operating system multi-task scheduling method according to an embodiment of this application. Figure 1 As shown, the method includes the following steps: Step S1: Collect the status information of each task in the operating system kernel in real time; The system obtains the status information of each task in real time from the operating kernel. The task status information includes task attributes and required system resource information, such as task priority, CPU usage, memory usage, and context switch count. This task status information can be recorded by the data structure task_state, which includes multiple features (fields) such as CPU usage, memory usage, context switch count, and task priority.

[0018] Step S2: The task state information is converted into a state vector and input into the reinforcement learning model for inference, so as to obtain the scheduling decision based on the State-Action table; Reinforcement learning models use a State-Action table to store the expected reward value of each possible action under different task states. The goal of reinforcement learning is to enable the system to gradually learn the optimal resource allocation strategy by continuously updating the State-Action table (expected reward value).

[0019] To this end, the Task_State space is set, including fields such as CPU utilization, GPU utilization, memory usage, task priority, and context switch count; the System_State space is set, including fields such as CPU core load, GPU load, and remaining memory; and the Scheduler_Action space is set to represent the scheduling decisions that the system can make, including fields such as adjusting task priority, setting CPU affinity, adjusting scheduling period, and modifying task memory allocation strategy.

[0020] Within the operating system kernel, task states can be transformed into a format suitable for indexing State-Action tables. For example, CPU utilization, representing task CPU usage, can be used as one of the table's indexes, combined with other task characteristics such as memory usage, context switch counts, and task priority to form a comprehensive state representation.

[0021] The multi-state features of each task are combined into a state vector, which serves as the input to the reinforcement learning model. Within the kernel, whenever a task needs to be scheduled, its current state is transformed into a state vector suitable for the reinforcement learning model to process, and then fed into the model for inference. This process occurs in kernel mode, where the reinforcement learning model makes scheduling decisions based on the current state.

[0022] In the State-Action table, each state corresponds to multiple actions, and the table stores the State-Action value for each action. At each time step, after the reinforcement learning model obtains the state vector, it selects the action with the largest State-Action value from the State-Action table based on the current task state. In other words, it selects the action with the largest State-Action value in the current state using a greedy strategy, such as adjusting task priority, setting CPU affinity, or adjusting memory allocation.

[0023] Step S3: Update the State-Action table based on the reward obtained from executing the scheduling decision; The goal of reinforcement learning is to adjust the State-Action value based on historical feedback (rewards). Each state corresponds to multiple actions, and a value table stores the State-Action value for each action. At each time step, the reinforcement learning algorithm selects the action with the highest State-Action value from the State-Action table based on the current task state. After the action is selected and executed, a reward is received. The reinforcement learning model updates the State-Action table to improve the task scheduling strategy, thereby continuously optimizing the task scheduling strategy and achieving intelligent resource allocation and task scheduling.

[0024] Suppose there are two tasks: Task 1 (obstacle avoidance task) and Task 2 (navigation task). Each task performs different scheduling actions under different states. The State-Action table is initialized as follows:

[0025] After a task is executed, a reward is evaluated based on the actual performance of each task. For example, task 1 (obstacle avoidance) may experience high CPU and GPU usage due to a surge in computational load, while task 2 (navigation) may have lower resource usage due to a reduced load. The system calculates the execution reward for each task based on a reward function and uses this reward to update the State-Action table.

[0026] As an example, the reward function assigns a reward value based on factors such as task execution efficiency, latency, and resource consumption. The specific calculation process includes: (1) Calculate the reward value based on the reward function; specifically, calculate the reward value determined by frame rate gain, energy consumption, and latency; reward = θ1 × fps_gain + θ2 × (1 - e_consumed) -θ3 × latency_penalty Where fps_gain is the frame rate gain value, e_consumed is the energy consumption value, and latency_penalty is the latency penalty value. θ1 controls the weight of frame rate gain (fps_gain), which is set to 0.5 in this example. θ2 controls the weight of energy consumption (e_consumed), which is set to 0.3 in this example. θ3 controls the weight of latency penalty (latency_penalty), which is set to 0.2 in this example.

[0027] (2) Adjust the reward value based on task status information; specifically, adjust the reward value based on factors such as task priority, CPU usage, GPU usage, and memory usage. For example, the following calculations are performed sequentially: Reward = reward + task_priority × E1, where task_priority represents the impact value corresponding to the task priority. For example, task priority can be divided into multiple levels, such as 10 levels, and the impact value is the corresponding level number; E1 represents the impact weight of the task priority, for example, 0.2. Using "+" in the calculation indicates that high-priority tasks are encouraged to be scheduled first.

[0028] reward = reward - cpu_uil × E2, where cpu_uil represents the impact value corresponding to CPU usage, for example, expressed as a percentage of CPU usage, and E2 represents the weight of CPU usage impact, for example, 0.1. Using "-" in the calculation indicates encouragement of low CPU usage.

[0029] reward = reward - gpu_util × E3, where gpu_util represents the impact value corresponding to GPU usage, for example, expressed as a percentage of GPU usage, and E3 represents the weight of GPU usage impact, for example, 0.1. Using "-" in the calculation indicates encouragement of low GPU usage.

[0030] reward = reward + mem_tuil × E4, where mem_tuil represents the impact value corresponding to memory usage, such as a percentage of memory usage, and E4 represents the weight of memory usage impact, such as 0.05. Using "+" in the calculation indicates encouragement of reasonable memory usage.

[0031] In this example, Task 1 (obstacle avoidance) requires real-time processing of large amounts of sensor data, such as LiDAR and depth camera data. It must be able to quickly identify obstacles and avoid them, thus placing a very high load on the CPU and GPU, resulting in a low fps_gain and a high e_consumed. Furthermore, it has very strict latency requirements, ensuring real-time processing for every frame; otherwise, collisions may occur, leading to a high latency_penalty. Therefore, in Task 1 (obstacle avoidance): fps_gain = 0.8, e_consumed = 0.9, latency_penalty = 0.2, cpu_util = 0.95, gpu_util = 0.9, mem_util = 0.8, task_priority = 10. The reward value obtained after the reward calculation function is 2.24.

[0032] Task 2 (navigation task), while also involving some computation (such as path planning and position correction), has less stringent real-time requirements than the obstacle avoidance task. Its computational load is lower, allowing for some latency tolerance without compromising safety. Therefore, its `fps_gain` is higher and its `e_consumed` is lower. The latency requirements for navigation tasks are relatively relaxed, resulting in a lower `latency_penalty`. Thus, in Task 2 (navigation task): `fps_gain = 1.1`, `e_consumed = 0.7`, `latency_penalty = 0.1`, `cpu_util = 0.6`, `gpu_util = 0.4`, `mem_util = 0.7`, `task_priority = 5`. The reward value obtained after the reward calculation function is 1.535.

[0033] Then, the State-Action value is updated using the following algorithm: State-Action[state][action] = (1 - α) × State-Action[state][action]+ α × (reward + γ × max_value_next) Here, α is the learning rate, representing the weight of newly acquired reward information in updating the State-Action value. It determines the degree of influence of new reward information on updating the State-Action value. A larger learning rate means the system adapts to new information more quickly, while a smaller learning rate means the system responds more slowly to new information and tends to rely more on previous experience. γ is a discount factor, representing the importance of the maximum future reward value `max_value_next` in the current decision. A γ close to 1 indicates that the system highly values ​​future rewards, while a γ close to 0 indicates that the system focuses more on current rewards. In this example, α is set to 0.5, and γ is set to 0.9.

[0034] Calculations show that the State-Action value for Task 1 (obstacle avoidance task) is updated to 1.4 when performing Action 1 (priority adjustment), and the State-Action value for Task 2 (navigation task) is updated to 1.1875 when performing Action 1 (priority adjustment). Based on these calculations, the final updated State-Action table is as follows:

[0035] Based on the selection strategy after learning the State-Action strength model, the system will select the task with the largest State-Action value for scheduling. Therefore, task 1 (obstacle avoidance task) will be scheduled first.

[0036] According to another embodiment of this application, a multi-task scheduling device for a real-time operating system based on reinforcement learning is also disclosed, comprising: The information acquisition module collects the status information of each task in the operating system kernel in real time; the task status information includes task attributes and required system resource information. The decision-making module transforms task state information into state vectors and inputs them into the reinforcement learning model for reasoning, so as to obtain scheduling decisions based on the State-Action table. The update module updates the State-Action table based on the rewards obtained from executing the scheduling decision; The reinforcement learning model uses a State-Action table to store the expected reward value for each possible action in different tasks, and selects the action with the highest expected reward value from the State-Action table to execute based on the current task. After task execution, the reinforcement learning model calculates the task's execution reward to update the State-Action table. The execution reward calculation includes: Calculate the reward value based on the reward function; The reward value is adjusted based on the task status information.

[0037] In traditional static priority scheduling, even if an obstacle avoidance task has high priority, if resources are already saturated, it may be affected by system resource contention, leading to low-priority tasks consuming too many resources and affecting the execution of high-priority tasks. In contrast, the intensity learning model provided in this application intelligently predicts the resource requirements of tasks and dynamically schedules them based on the actual load of the current tasks, ensuring that high-priority tasks (such as the obstacle avoidance task in the example above) receive sufficient resources, even if the system load increases due to terrain complexity. If system resources are particularly scarce, the intensity learning strategy can even adjust task priorities to ensure the real-time performance of critical tasks.

[0038] Therefore, this application employs a dynamic scheduling algorithm based on reinforcement learning and State-Action management. This algorithm ensures that high-priority tasks receive priority scheduling while also dynamically adjusting based on real-time task load and resource consumption. Furthermore, a State-Action table manages the State-Action values ​​of tasks in different states, and the table data is updated through a reinforcement learning model, guaranteeing the adaptive learning capability of tasks in multi-task scheduling.

[0039] Furthermore, by incorporating task priorities into the State-Action value update process, high-priority tasks are ensured to be scheduled even when resources are scarce; the reward weighting of high-priority tasks enables them to obtain sufficient resources in a multi-tasking environment, thereby improving task execution efficiency.

[0040] Furthermore, the system dynamically adjusts task scheduling based on the usage of resources such as CPU, GPU, and memory, avoiding resource contention and task conflicts in traditional static scheduling algorithms. The system can adaptively adjust the scheduling strategy in resource-scarce or complex environments to ensure the real-time performance and priority of task execution.

[0041] Furthermore, the reward function designed in this application calculates the task reward by comprehensively considering frame rate gain, energy consumption, latency penalty, resource usage, and task priority, which can dynamically optimize the task execution order.

[0042] This application utilizes reinforcement learning to achieve dynamic task scheduling, combining intelligent adjustments to task priority and resource consumption, resulting in highly flexible and real-time performance. Compared to traditional static priority and rule-based scheduling algorithms, this application offers significant advantages in multi-task scheduling and real-time task management.

[0043] Some deep learning-based task scheduling algorithms configured on cloud servers typically require significant computing resources and involve complex model training, making them unsuitable for real-time embedded or mobile environments. In contrast, the state-action-managed intensity learning model and method presented in this application offer significant advantages in terms of real-time performance, computational overhead, and system complexity. Therefore, they are particularly suitable for embedded systems and mobile operating systems, such as drones and robots, providing flexible real-time scheduling capabilities to the operating system through deep learning algorithms.

[0044] Although this application has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications can still be made to the specific implementation of this application or equivalent substitutions can be made to some technical features without departing from the spirit of the technical solution of this application, and all such modifications and substitutions should be covered within the scope of the technical solution claimed in this application.

Claims

1. A multi-task scheduling method for a real-time operating system based on reinforcement learning, characterized in that, include: Real-time acquisition of status information for each task in the operating system kernel; the task status information includes task attributes and required system resource information; The task state information is transformed into a state vector and input into the reinforcement learning model for inference, so as to obtain scheduling decisions based on the State-Action table; The State-Action table is updated based on the reward obtained from executing the scheduling decision; The reinforcement learning model uses a State-Action table to store the expected reward value for each possible action in different tasks, and selects the action with the highest expected reward value from the State-Action table to execute based on the current task. After task execution, the reinforcement learning model calculates the task's execution reward to update the State-Action table. The execution reward calculation includes: Calculate the reward value based on the reward function; The reward value is adjusted based on the task status information.

2. The real-time operating system multi-task scheduling method according to claim 1, characterized in that, The reward function is: reward = θ1 × fps_gain + θ2 × (1 - e_consumed) -θ3 × latency_penalty Where reward is the reward value, fps_gain is the frame rate gain value, e_consumed is the energy consumption value, latency_penalty is the latency penalty value, and θ1, θ2, and θ3 are the weights.

3. The real-time operating system multi-task scheduling method according to claim 2, characterized in that, The task attributes include task priority, and the system resource information includes one or more of CPU usage, GPU usage, memory usage, and context switching count; the reward value is adjusted based on the task priority and one or more of the system resource information.

4. The real-time operating system multi-task scheduling method according to claim 3, characterized in that, Adjusting reward values ​​based on task priority includes increasing the reward value by multiplying the task priority's impact value by its weight; adjusting reward values ​​based on system resource information includes decreasing the reward value by multiplying the resource usage's impact value by its weight.

5. The real-time operating system multi-task scheduling method according to claim 3, characterized in that, The actions stored in the State-Action table include adjusting task priority, setting CPU affinity, adjusting scheduling cycle, modifying task memory allocation, and not making any adjustments.

6. The real-time operating system multi-task scheduling method according to claim 3, characterized in that, The State-Action values ​​in the State-Action table are updated using the following method: State-Action[state][action] = (1 - α) × State-Action[state][action] + α× (reward + γ × max_value_next) Where State-Action[state][action] is the State-Action value corresponding to the task state and action, α is the learning rate, γ is the discount factor, and max_value_next represents the maximum future reward value.

7. A multi-task scheduling device for a real-time operating system based on reinforcement learning, characterized in that, include: The information acquisition module collects the status information of each task in the operating system kernel in real time; the task status information includes task attributes and required system resource information. The decision-making module transforms task state information into state vectors and inputs them into the reinforcement learning model for reasoning, so as to obtain scheduling decisions based on the State-Action table. The update module updates the State-Action table based on the rewards obtained from executing the scheduling decision; The reinforcement learning model uses a State-Action table to store the expected reward value for each possible action in different tasks, and selects the action with the highest expected reward value from the State-Action table to execute based on the current task. After task execution, the reinforcement learning model calculates the task's execution reward to update the State-Action table. The execution reward calculation includes: Calculate the reward value based on the reward function; The reward value is adjusted based on the task status information.

8. The real-time operating system multi-task scheduling device according to claim 7, characterized in that, The reward function is: reward = θ1 × fps_gain + θ2 × (1 - e_consumed) -θ3 × latency_penalty Where reward is the reward value, fps_gain is the frame rate gain value, e_consumed is the energy consumption value, latency_penalty is the latency penalty value, and θ1, θ2, and θ3 are the weights.

9. The real-time operating system multi-task scheduling device according to claim 8, characterized in that, The task attributes include task priority, and the system resource information includes one or more of CPU usage, GPU usage, memory usage, and context switching count; the reward value is adjusted based on the task priority and one or more of the system resource information.

10. The real-time operating system multi-task scheduling device according to claim 9, characterized in that, The actions stored in the State-Action table include adjusting task priority, setting CPU affinity, adjusting scheduling cycle, modifying task memory allocation, and not making any adjustments.

Images