Task scheduling method and device based on reinforcement learning, medium and product

Abstract

The invention discloses a task scheduling method and device based on reinforcement learning, a medium and a product, and relates to the technical field of computers, and the method comprises the steps: determining a task scheduling action of a current time step according to a current action empirical value table and a preset scheduling constraint condition; updating the current system environment information and the current task load information of the embedded operating system based on the current reward information after the embedded operating system executes the task scheduling action; and based on the current system environment information, the current task load information and the current reward information, updating an empirical value in the current action empirical value table, and skipping to the step of determining the task scheduling action of the current time step according to the current action empirical value table and a preset scheduling constraint condition, and executing the task until the target task is scheduled to the target processor core meeting the preset condition. Tasks can be accurately allocated to appropriate hardware resources, and idle or overuse of the resources is avoided.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a task scheduling method, device, medium and product based on reinforcement learning. Background Art

[0002] In the Linux kernel operating system, task scheduling allocation on a specific processor hardware architecture (multiple CPU cores share one L2 cache) can improve cache utilization. However, when multiple cores access the L2 shared cache at the same time, it also brings the risk of over-caching, which may lead to cache performance degradation.

[0003] In the prior art, under a specific processor hardware architecture, when there is a task process, the scheduler schedules the L3 cache to achieve load balancing, or the scheduler preferentially schedules the process to a completely idle scheduling domain so that the process can use the entire L2 shared cache in the scheduling domain. These two technologies can only refine the task scheduling within the original CPU node range to the granularity of the scheduling domain.

[0004] In summary, under a specific processor hardware architecture, how to refine task scheduling to a specific CPU core is a technical problem to be solved in this field. Summary of the invention

[0005] The purpose of the embodiments of the present invention is to provide a task scheduling method, device, medium and product based on reinforcement learning, which can solve the task scheduling problem of processor cores sharing L2 cache.

[0006] In a first aspect, the present invention discloses a task scheduling method based on reinforcement learning, comprising:

[0007] Determine the task scheduling action of the current time step according to the current action experience value table and the preset scheduling constraint conditions; the task scheduling action includes a first action, a second action and a third action determined based on the target experience value corresponding to the current time step in the current action experience value table; the first action is used to select a target processor node for executing the target task, the second action is used to select a target scheduling domain in the target processor node, and the third action is used to select a processor core in the target scheduling domain; each scheduling domain is configured with a number of processor cores that can share a preset secondary cache;

[0008] Based on the current reward information after the embedded operating system executes the task scheduling action, the current system environment information and the current task load information of the embedded operating system are updated;

[0009] Based on the current system environment information, the current task load information and the current reward information, the experience value in the current action experience value table is updated, and the step of determining the task scheduling action of the current time step according to the current action experience value table and the preset scheduling constraints is jumped to, until the target task is scheduled to the target processor core that meets the preset conditions for task execution.

[0010] Optionally, based on the current system environment, the current task load information, and the current reward information, before updating the experience value in the current action experience value table, the following is also included:

[0011] Obtain the current processor hardware architecture information and current task information of the embedded operating system to obtain the current system environment information;

[0012] The current hardware utilization status information and current task running status information of the embedded system are obtained to obtain the current task load information.

[0013] Optionally, before determining the task scheduling action of the current time step according to the current action experience value table and the preset scheduling constraint condition, the method further includes:

[0014] The first affinity rule of the processor node, the second affinity rule of the scheduling domain, and the third affinity rule of the processor core in the processor hardware architecture information are set hierarchically based on the preset scheduling target; the preset scheduling target is a scheduling target determined based on having the best system performance in the scheduling process;

[0015] The preset scheduling constraint conditions of the scheduling task process are constructed based on the first affinity rule, the second affinity rule and the third affinity rule.

[0016] Optionally, the task scheduling action of the current time step is determined according to the current action experience value table and preset scheduling constraints, including:

[0017] Determine a processor node selection action experience value corresponding to the current time step in the current action experience value table, so as to select a target processor node for executing the target task that satisfies a first affinity rule from local multi-processor nodes based on the processor node selection action experience value;

[0018] Determine a scheduling domain selection action experience value corresponding to the current time step in the current action experience value table, so as to select a target scheduling domain that satisfies the second affinity rule from each scheduling domain in the target processor node based on the scheduling domain selection action experience value;

[0019] The processor core selection action experience value corresponding to the current time step in the current action experience value table is determined, so as to select a processor core that satisfies the third affinity rule from each processor core in the scheduling domain based on the processor core selection action experience value.

[0020] Optionally, the first affinity rule of the processor node, the second affinity rule of the scheduling domain, and the third affinity rule of the processor core in the processor hardware architecture information are set hierarchically based on the preset scheduling target, including:

[0021] A processor node utilization threshold in the processor hardware architecture information is set based on a preset scheduling target, so as to construct a first affinity rule of the task scheduling process according to a size relationship between a current processor node utilization and the processor node utilization threshold;

[0022] A second affinity rule is set based on a preset scheduling target to schedule the execution task threads of the same task and the process tasks with data sharing to the same scheduling domain;

[0023] A third affinity rule is set based on a preset scheduling target to schedule the task to a processor core with the lowest utilization rate in the scheduling domain.

[0024] Optionally, obtain the current processor hardware architecture information and current task information of the embedded system, including:

[0025] Acquire the associated information of the hardware information of the current system environment of the embedded operating system to obtain the current processor hardware architecture information constructed based on the physical structures of the processor and the interconnection information of the physical structures and the preset secondary cache, wherein the physical structure includes a processor node, a scheduling domain, and a processor core;

[0026] Obtain the task priority, task cycle information, execution task thread, and data dependency between tasks of the current task in the embedded operating system to obtain the corresponding current task information.

[0027] Optionally, before obtaining the current processor hardware architecture information and current task information of the embedded operating system, the following steps are also included:

[0028] The processor cores on the embedded operating system are divided according to whether they share a preset secondary cache to obtain a scheduling domain;

[0029] Each scheduling domain is divided according to whether the local memory of the same node is accessed to obtain the corresponding processor node;

[0030] The physical structure of the processor is determined based on the architectural relationship between the processor node, scheduling domain, and processor core.

[0031] Optionally, obtain the current hardware utilization status information of the embedded system, including:

[0032] The utilization rate of the processor core and the usage of the shared preset secondary cache in the embedded operating system during the current system operation are obtained to obtain the corresponding current hardware utilization status information.

[0033] Optionally, determining a processor node selection action experience value corresponding to the current time step in the current action experience value table, and selecting a target processor node for executing the target task that satisfies the first affinity rule from the local multi-processor nodes based on the processor node selection action experience value, includes:

[0034] Determine the maximum processor node selection action experience value in the current action experience value table, and select a processor node whose current processor node utilization is less than a processor node utilization threshold and has a task load from the local multi-processor nodes based on the maximum processor node selection action experience value as the target processor node for executing the target task.

[0035] Optionally, selecting a processor node whose current processor node utilization is less than a processor node utilization threshold and has a task load from the local multi-processor nodes to determine as a target processor node for executing the target task includes:

[0036] Obtaining the current processor node utilization of each processor node recorded in the task load information;

[0037] Selecting a processor node whose current processor node utilization is less than a processor node utilization threshold as an initial processor node;

[0038] Determine whether there is a processor node with existing task load among the initial processor nodes;

[0039] If there is only one processor node with a task load among the initial processor nodes, the processor node with the task load is determined as the target processor node for executing the target task.

[0040] Optionally, after determining whether there is a processor node with an existing task load in the initial processor node, the method further includes:

[0041] If there are multiple processor nodes with existing task loads among the initial processor nodes, the processor node with the existing task load that is closest in physical distance is determined as the target processor node for executing the target task.

[0042] Optionally, the task scheduling method based on reinforcement learning of the present application further includes:

[0043] Based on the values ​​of each state parameter in the state parameter variation range generated by each scheduling action, determine the task scheduling action that can be executed at the current time step to construct a target action set; the state parameter variation range is the state parameter variation range corresponding to the current system environment information and the current task load information at the current time step;

[0044] Obtain the experience value corresponding to each action in the target action set from the current action experience value table to obtain the experience value set corresponding to the current time step;

[0045] According to the highest value selection rule, the target experience value corresponding to the current time step is determined from the experience value set.

[0046] In a second aspect, the present invention discloses an electronic device, comprising:

[0047] Memory, used to store computer programs;

[0048] A processor is used to execute a computer program to implement the steps of the aforementioned disclosed task scheduling method based on reinforcement learning.

[0049] In a third aspect, the present invention discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the steps of the aforementioned disclosed reinforcement learning-based task scheduling method are implemented.

[0050] In a fourth aspect, the present invention discloses a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the aforementioned reinforcement learning-based task scheduling method disclosed above.

[0051] It can be seen that the present invention discloses a task scheduling method based on reinforcement learning, including: determining the task scheduling action of the current time step according to the current action experience value table and the preset scheduling constraint condition; the task scheduling action includes the first action, the second action and the third action determined based on the target experience value corresponding to the current time step in the current action experience value table; the first action is used to select the target processor node to execute the target task, the second action is used to select the target scheduling domain in the target processor node, and the third action is used to select the processor core in the target scheduling domain; each scheduling domain is configured with a plurality of processor cores that can share the preset secondary cache; based on the current reward information after the embedded operating system executes the task scheduling action, the current system environment information and the current task load information of the embedded operating system are updated; based on the current system environment information, the current task load information and the current reward information, the experience value in the current action experience value table is updated, and the step of determining the task scheduling action of the current time step according to the current action experience value table and the preset scheduling constraint condition is jumped to until the target task is scheduled to the target processor core that meets the preset conditions for task execution.

[0052] It can be seen from the above technical solutions that by comprehensively considering the system environment information and task load information, and through reinforcement learning, tasks can be accurately assigned to appropriate hardware resources, avoiding idle or overuse of resources, taking into account the task scheduling optimization problem on the processor architecture with preset secondary cache, and avoiding the degradation of application performance caused by cache contention. According to the task association relationship and task load information of the target processor node, the target task is scheduled to run on the processor core with the lowest utilization in the appropriate scheduling domain. The data sharing advantage brought by the shared preset secondary cache can be fully utilized to reduce the delay of data access during task execution, thereby speeding up the execution speed of tasks, improving the execution efficiency of single tasks and the overall performance of the system for multi-tasking. The core selection scheduling of ready tasks is optimized to achieve refined scheduling at the CPU core granularity, maximizing the guarantee of the running performance of application tasks. Finally, with the operation of the embedded operating system, tasks are constantly generated and terminated, and the system environment and task load conditions are in dynamic changes. By updating the system environment information and task load information, these changes can be tracked in real time, so that subsequent task scheduling decisions can always be adjusted based on the latest and accurate system status. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0054] Figure 1 A flow chart of a task scheduling method based on reinforcement learning provided by an embodiment of the present invention;

[0055] Figure 2 A module diagram of a task scheduling system based on reinforcement learning provided by an embodiment of the present invention;

[0056] Figure 3 A flowchart of a specific task scheduling method based on reinforcement learning provided by an embodiment of the present invention;

[0057] Figure 4 A structural diagram of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] The terms "including" and "having" in the specification of the present invention and the above-mentioned drawings, as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but may include steps or units that are not listed.

[0060] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0061] In the Linux kernel operating system, task scheduling allocation on a specific processor hardware architecture (multiple CPU cores share one L2 cache) can improve cache utilization. However, when multiple cores access the L2 shared cache at the same time, it also brings the risk of over-caching, which may lead to cache performance degradation.

[0062] In the prior art, under a specific processor hardware architecture, when there is a task process, the scheduler schedules the L3 cache to achieve load balancing, or the scheduler preferentially schedules the process to a completely idle scheduling domain so that the process can use the entire L2 shared cache in the scheduling domain. These two technologies can only refine the task scheduling within the original CPU node range to the granularity of the scheduling domain.

[0063] To this end, the present invention provides a task scheduling solution based on reinforcement learning, which can refine task scheduling to specific CPU cores under a specific processor hardware architecture.

[0064] Reference Figure 1 As shown, the present invention discloses a task scheduling method based on reinforcement learning, comprising:

[0065] Step S11: determine the task scheduling action of the current time step according to the current action experience value table and the preset scheduling constraints; the task scheduling action includes a first action, a second action and a third action determined based on the target experience value corresponding to the current time step in the current action experience value table; the first action is used to select a target processor node to execute the target task, the second action is used to select a target scheduling domain in the target processor node, and the third action is used to select a processor core in the target scheduling domain; each scheduling domain is configured with a number of processor cores that can share a preset secondary cache.

[0066] In this embodiment, before determining the task scheduling action of the current time step according to the current action experience value table and the preset scheduling constraints, it also includes: setting the first affinity rule of the processor node in the processor hardware architecture information, the second affinity rule of the scheduling domain, and the third affinity rule of the processor core in a hierarchical manner based on the preset scheduling target; the preset scheduling target is a scheduling target determined based on the optimal system performance in the scheduling process; and constructing the preset scheduling constraints of the scheduling task process based on the first affinity rule, the second affinity rule, and the third affinity rule. It can be understood that before determining the task scheduling action of the current time step, the affinity rules are set hierarchically according to the scheduling target in advance to obtain the preset scheduling constraints constructed by the affinity rules of different levels. After determining the L2 cache architecture, the affinity of task scheduling at three levels is constructed, namely, the affinity on the central processor node, the affinity on the scheduling domain granularity, and the affinity on each central processor core within the scheduling domain. When the goal is to optimize system performance, the affinity rules for scheduling tasks on the CPU core are: 1) Ensure that tasks with shared data are scheduled in the shared preset L2 cache domain to avoid performance loss caused by cache jitter; 2) When there are multiple CPU nodes or CPU cores to choose from, select the node / core with the lowest utilization to avoid performance loss caused by CPU preemption. The affinity rules are set as follows:

[0067] A processor node utilization threshold in the processor hardware architecture information is set based on a preset scheduling target to construct a first affinity rule for the task scheduling process based on the size relationship between the current processor node utilization and the processor node utilization threshold; a second affinity rule is set based on the preset scheduling target to schedule the execution task threads of the same task and process tasks with data sharing to the same scheduling domain; a third affinity rule is set based on the preset scheduling target to schedule the task to the processor core with the lowest utilization in the scheduling domain.

[0068] It can be understood that for the central processing unit node affinity, that is, the first affinity rule of the CPU node, for a system with multiple CPU nodes in the system, the utilization upper limit of the CPU node (the average utilization on all cores of the CPU node) is first set. If it is lower than this processor node utilization threshold, the task is scheduled to the same CPU node, otherwise it is scheduled to other CPU nodes.

[0069] For the affinity within the scheduling domain, since multiple CPU cores (these cores are called a scheduling domain) within each scheduling domain share an L2 cache, it is necessary to set a second affinity rule for task scheduling to prevent performance loss caused by tasks competing for the L2 cache during operation. These second affinity rules constitute the constraints for task scheduling based on reinforcement learning. They include the following situations: 1) Threads under the same task need to be scheduled to a cluster that shares an L2 cache; 2) Processes that share data need to be scheduled to a cluster that shares an L2 cache; 3) Processes with mutually exclusive access are scheduled to different clusters; 4) Other tasks are scheduled to low-load clusters based on CPU utilization.

[0070] For the third affinity rule on each CPU core, first set the utilization upper limit of each CPU core and schedule the task to the CPU core with the lowest current utilization within the cluster.

[0071] In this embodiment, the task scheduling action of the current time step is determined based on the reinforcement learning algorithm and according to the current action experience value table and the preset scheduling constraints, wherein the task scheduling action of the current time step specifically includes the first action, the second action and the third action. It should be noted that the selection of the first action, the second action and the third action obtains the target experience value at the current time step from the current action experience value table, and is under the constraint of the preset scheduling constraints, with the scheduling action corresponding to the target experience value as the task scheduling action of the current time step.

[0072] Specifically, based on the values ​​of each state parameter in the state parameter variation range generated by each scheduling action, the task scheduling action executable at the current time step is determined to construct a target action set; the state parameter variation range is the state parameter variation range corresponding to the current system environment information and the current task load information of the current time step; the experience value corresponding to each action in the target action set is obtained from the current action experience value table to obtain the experience value set corresponding to the current time step; the target experience value corresponding to the current time step is determined from the experience value set according to the highest value selection rule. It can be understood that according to the values ​​of each state parameter generated by each scheduling action, the task scheduling action executable at the current time step is determined, wherein each scheduling action can specifically include: the action of scheduling tasks to processor nodes, the action of scheduling tasks to scheduling domains, and the action of scheduling tasks to processor cores. It can be understood that the action space is defined so that the task selects a CPU node on multiple CPU nodes, a Custer scheduling domain on the CPU node, and a specific CPU core on the Cluster scheduling domain as the platform for task execution during operation. According to the preset scheduling constraints defined in the present invention, the incentive function includes three types of operations in the action space: 1) When selecting CPU node scheduling, the incentive for each selection of a CPU node is to ensure that the system uses as few CPU nodes as possible under the premise of not exceeding the set CPU node utilization threshold (the threshold indicates that when the CPU node is lower than the upper limit, the task execution will not be blocked). If there are multiple CPU nodes that meet the conditions in the system, the node with the closest physical distance to the already loaded CPU node is preferred, because usually the closer the distance between two CPU nodes is, the faster the speed is when accessing each other's memory; 2) When selecting a scheduling domain, the incentive for each selection of a scheduling domain is to ensure that threads within the same task and multiple tasks with shared data are scheduled to the same scheduling domain, and multiple tasks with mutually exclusive data access are scheduled to different scheduling domains; 3) When selecting a specific CPU core, the incentive for each selection of a CPU core is to ensure that the CPU cores used in the scheduling domain are as few as possible under the premise of not exceeding the set CPU core utilization threshold.

[0073] Specifically, the action of scheduling tasks to processor nodes includes: the scheduling action of scheduling tasks to a CPU node labeled 1, the scheduling action of scheduling tasks to a CPU node labeled 2, ..., the scheduling action of scheduling tasks to a CPU node labeled N, where N represents the total number of CPU nodes in the current embedded system; the action of scheduling tasks to scheduling domains includes: the scheduling action of scheduling tasks from a CPU node to a scheduling domain labeled 1.1, the scheduling action of scheduling tasks from a CPU node to a scheduling domain labeled 1.2, ..., the scheduling action of scheduling tasks from a CPU node to a scheduling domain labeled M, where M represents the total number of scheduling domains; the action of scheduling tasks to processor cores includes: the scheduling action of scheduling tasks from a certain scheduling domain to an internal CPU core labeled 2.1, the scheduling action of scheduling tasks from a certain scheduling domain to an internal CPU core labeled 2.2, ..., the scheduling action of scheduling tasks from a certain scheduling domain to an internal CPU core labeled 2.Z, where Z represents the total number of CPU cores in the same scheduling domain. In this way, the target action set can be constructed, and based on the experience value corresponding to each scheduling action in the target action set recorded in the current action experience value table, the experience value set at the current time step is obtained, and finally the target experience value corresponding to the highest value in the experience value set is selected.

[0074] In this embodiment, the processor node selection action experience value corresponding to the current time step in the current action experience value table is determined, so as to select a target processor node for executing the target task that satisfies the first affinity rule from the local multi-processor node based on the processor node selection action experience value; the scheduling domain selection action experience value corresponding to the current time step in the current action experience value table is determined, so as to select a target scheduling domain that satisfies the second affinity rule from each scheduling domain in the target processor node based on the scheduling domain selection action experience value; the processor core selection action experience value corresponding to the current time step in the current action experience value table is determined, so as to select a processor core that satisfies the third affinity rule from each processor core in the scheduling domain based on the processor core selection action experience value. It can be understood that after determining the selected target experience value, the execution action (first action, second action and third action) corresponding to the target experience value is determined, and when executing the first action, the following steps are specifically performed:

[0075] Determine the maximum processor node selection action experience value in the current action experience value table, and select a processor node whose current processor node utilization is less than the processor node utilization threshold and has a task load from the local multi-processor nodes based on the maximum processor node selection action experience value to determine it as the target processor node for executing the target task. It can be understood that when there is a target task, each CPU node is determined based on the current processor hardware architecture information of the system environment information, and based on the experience value selected in the current action experience value table, the CPU node recorded in the task load information that meets the above definition and conditions is further determined as the target CPU node for executing the target task.

[0076] Specifically, the current processor node utilization of each processor node recorded in the task load information is obtained; the processor node whose current processor node utilization is less than the processor node utilization threshold is selected as the initial processor node; it is determined whether there is a processor node with an existing task load in the initial processor node; if there is only one processor node with an existing task load in the initial processor node, the processor node with the existing task load is determined as the target processor node for executing the target task; if there are multiple processor nodes with existing task loads in the initial processor node, the processor node with the existing task load closest in physical distance is determined as the target processor node for executing the target task. It can be understood that according to the action definition, when selecting a CPU node, if there is a CPU node whose current processor node utilization is less than the processor node utilization threshold, the CPU node that meets the condition is used as the initial processor node, and then it is further determined whether there is a CPU node with an existing task load in the initial CPU node. If there is one and only one CPU node that meets this condition in the initial CPU node, the initial CPU node is directly used as the target CPU node for executing the target task. If there are multiple CPU nodes that meet this condition in the initial CPU node, the CPU node with the closest physical distance is preferentially selected as the target CPU node to save the time of accessing each other's memory.

[0077] Specifically, the algorithm for selecting the CPU node, scheduling domain, and CPU core is the Q-learning reinforcement learning algorithm, the first task scheduling work is performed, and the corresponding current reward information is obtained through a preset incentive function, wherein the Q-learning reinforcement learning algorithm used in the present invention is as follows:

[0078] ;

[0079] in, , represent the learning rate and discount factor respectively. It is used to control the amplitude of each Q value update and determines the speed of algorithm convergence; is a discount factor with a value between 0 and 1; It means that at the time step of In this state, take task scheduling action The expected value of the benefit can be obtained, that is, the Q value. The system environment will feedback the corresponding incentives according to the task scheduling action. Therefore, the main idea of ​​the algorithm is to construct the state space and action space into a two-dimensional table to store the Q value, and then combine the Q value to select the task scheduling action that can obtain the maximum incentive.

[0080] Step S12: Based on the current reward information after the embedded operating system executes the task scheduling action, the current system environment information and the current task load information of the embedded operating system are updated.

[0081] In this embodiment, after executing the task scheduling action, the running information generated is used for incentive feedback, that is, the current system environment information and the current task load information of the embedded operating system are updated according to the returned running information. In the process of updating the information, the corresponding information of the system environment information and the task load information that generates state fluctuations is mainly updated, while the static information therein does not need to be updated.

[0082] Step S13: Based on the current system environment information, the current task load information and the current reward information, the experience value in the current action experience value table is updated, and the process jumps to the step of determining the task scheduling action of the current time step according to the current action experience value table and the preset scheduling constraints, until the target task is scheduled to the target processor core that meets the preset conditions for task execution.

[0083] In this embodiment, based on the current system environment information, the current task load information and the current reward information, before updating the experience value in the current action experience value table, it also includes: obtaining the current processor hardware architecture information and the current task information of the embedded system to obtain the current system environment information; obtaining the current hardware utilization status information and the current task running status information of the embedded system to obtain the current task load information. It can be understood that before task scheduling, it is first necessary to obtain the current system environment and the current task load information to construct the environment and state in reinforcement learning, which includes two types of static information and runtime dynamic information.

[0084] Static information includes hardware information and task information in the system. For the hardware information part, that is, the processor hardware architecture information, it shows the physical structure of the processor. Therefore, when obtaining the current system environment information and the current task load information, it includes: obtaining the associated information of the hardware information of the current system environment of the embedded operating system to obtain the current processor hardware architecture information based on the physical structures of the processor and the interconnection information of each physical structure and the preset secondary cache, wherein the physical structure includes processor nodes, scheduling domains, and processor cores; obtaining the task priority, task cycle information, execution task thread, and data dependencies between tasks of the current task in the embedded operating system to obtain the corresponding current task information.

[0085] Before obtaining the current processor hardware architecture information, it also includes: dividing the processor cores on the embedded operating system according to whether they share the preset secondary cache to obtain scheduling domains; dividing each scheduling domain according to whether they access the local memory of the same node to obtain the corresponding processor nodes; determining the physical structure of the processor according to the architectural relationship between the processor nodes, scheduling domains, and processor cores.

[0086] It is understandable that the processor hardware architecture information includes which processor nodes the processor chip is composed of, which scheduling domains each processor node contains, and which CPU cores each scheduling domain contains; it also includes the interconnection information between these hardware components and the L2 cache, for example: which CPU cores share an L2 cache. For task information, it describes information such as task priority and periodicity, which threads the task is composed of, and whether the task has data dependencies with other tasks in the system.

[0087] In this embodiment, obtaining the current hardware utilization status information of the embedded system includes: obtaining the utilization rate of the processor core and the usage of the shared preset secondary cache in the embedded operating system when the current system is running, so as to obtain the corresponding current hardware utilization status information. It can be understood that the dynamic information includes hardware utilization status information and task running status information. The hardware utilization status information mainly describes the current utilization rate of the central processing unit core and the usage of the L2 cache; the task running status information mainly describes the current state of the task, including information such as the ready state and the running state.

[0088] In this way, the experience value in the current action experience value table is updated by obtaining the current system environment information, the current task load information and the current reward information, so that the action experience value table is updated with the experience value according to each execution of the action, so that when the next execution action is selected, the experience value of the scheduling action of the optimal processor node, the optimal scheduling domain, and the optimal processor core can be determined from the current action experience value table, and the corresponding scheduling action can be selected to execute until the selected CPU core meets the termination requirements.

[0089] like Figure 2 As shown, a module structure diagram of a preset task scheduling system is disclosed, wherein: Figure 2 For the specific implementation process of each module, please refer to the aforementioned public content and will not be repeated here.

[0090] In order to further improve the overall energy efficiency of the system, when the system receives the target task, the target task is analyzed in advance for its energy consumption characteristics to determine whether the target task's energy consumption type is a low energy consumption type or a high energy consumption type. The specific judgment criteria can be set according to historical data, and no further description is given. When the energy consumption type of the target task is determined to be a low energy consumption type, when selecting the target CPU node, in addition to selecting the node according to the above scheduling conditions and load information, the scheduling conditions for the low energy consumption core conditions of each core in the CPU node are added. In this way, when scheduling tasks, low energy consumption tasks are preferentially assigned to low power consumption CPU cores. When the energy consumption type of the target task is determined to be a high energy consumption type, when selecting the target CPU node, in addition to selecting the node according to the above scheduling conditions and load information, the scheduling conditions for the high energy consumption core conditions of each core in the CPU node are added. In this way, when scheduling tasks, high energy consumption tasks are preferentially assigned to high power consumption CPU cores, optimizing the overall energy consumption of the system. For example, some simple sensor data collection tasks (low-power target tasks) are scheduled to the CPU core running in energy-saving mode; while for computationally intensive tasks that are not very sensitive to energy consumption, such as scientific computing simulation tasks, they can be assigned to high-performance but higher-energy-consuming cores, thereby optimizing the overall energy consumption of the system.

[0091] Reference Figure 3As shown, according to the defined action space and incentive function, the execution process of the task scheduling method based on reinforcement learning is as follows: First, obtain the system environment information and task load information, and establish the corresponding Q-learning reinforcement learning algorithm according to the action space on the core selection scheduling and the state space on the system environment model. For scheduling at three levels, set the utilization threshold on the CPU node and each CPU core. Based on the reinforcement learning algorithm, first select the optimal CPU node and schedule the task to the node, then select the optimal Cluster and the CPU core in the Cluster in turn, and finally accurately schedule the task to run on a specific CPU core. Specifically, this embodiment takes a platform with a 2-CPU node architecture as an example to illustrate the specific implementation process of the method proposed in the present invention. Each CPU node has 4 Cluster scheduling domains, and each Cluster has 32 CPU cores. These cores share an L2 cache. The implementation process is as follows:

[0092] 1) Obtain system environment information, including CPU, Cluster, CPU core information and cache information, and set utilization thresholds for each CPU node and CPU core. For example, set the utilization limit of each CPU node to 50% and the utilization limit of each core to 60%.

[0093] 2) Obtain task load information. When there are two tasks A and B that share data in the system, the main information is the task status (whether it is ready), the scheduling core, and the relationship with other tasks. Here, task A is set to the ready state, the scheduling core is NULL, and it shares data with task B; task B is set to the running state, the scheduling core is core 0 on cluster 0 on CPU 0, and the utilization rate of core 0 is 10%.

[0094] 3) First, perform the optimization operation on the CPU node through reinforcement learning methods (such as Q-learning). The average utilization of CPU 0 in the current system is calculated to be 3%, and task A also selects CPU 0; if there is no suitable node, repeat step 3).

[0095] 4) Perform optimization operations on the cluster based on Q-learning. Task B, which shares data with Task A, is currently running on Cluster 0, so Task A also chooses Cluster 0;

[0096] 5) If no suitable cluster is found in other cases, first repeat step 4). If still not successful, fall back to repeat step 3). At this time, you need to select another CPU node.

[0097] 6) Perform optimization operations on CPU cores based on Q-learning. Task B is currently running on core 0, and the current core utilization is 10%, which is lower than 60%, so task A also chooses core 0;

[0098] 7) If no suitable core is found in other cases, first repeat step 6). If still unsuccessful, fall back to repeat step 4). At this time, you need to select another Cluster.

[0099] 8) After the core is selected, run Task A and update the status of Task A to Running. The scheduling core is (0-0-0), that is, CPU 0, Cluster 0, and Core 0. At the same time, update the resource utilization on the CPU cores and nodes in the system.

[0100] In this way, for the processor architecture with shared L2 cache, accurate task scheduling on the CPU core for performance optimization is achieved based on real-time perception of the system environment and task load, fully considering the shared data characteristics between tasks to avoid performance losses caused by shared L2 cache; at the same time, the CPU node and core utilization are considered in task scheduling to reduce performance losses caused by CPU overload.

[0101] It can be seen that the present invention discloses a task scheduling method based on reinforcement learning, including: determining the task scheduling action of the current time step according to the current action experience value table and the preset scheduling constraint condition; the task scheduling action includes the first action, the second action and the third action determined based on the target experience value corresponding to the current time step in the current action experience value table; the first action is used to select the target processor node to execute the target task, the second action is used to select the target scheduling domain in the target processor node, and the third action is used to select the processor core in the target scheduling domain; each scheduling domain is configured with a plurality of processor cores that can share the preset secondary cache; based on the current reward information after the embedded operating system executes the task scheduling action, the current system environment information and the current task load information of the embedded operating system are updated; based on the current system environment information, the current task load information and the current reward information, the experience value in the current action experience value table is updated, and the step of determining the task scheduling action of the current time step according to the current action experience value table and the preset scheduling constraint condition is jumped to until the target task is scheduled to the target processor core that meets the preset conditions for task execution.

[0102] It can be seen from the above technical solutions that by comprehensively considering the system environment information and task load information, and through reinforcement learning, tasks can be accurately assigned to appropriate hardware resources, avoiding idle or overuse of resources, taking into account the task scheduling optimization problem on the processor architecture with preset secondary cache, and avoiding the degradation of application performance caused by cache contention. According to the task association relationship and task load information of the target processor node, the target task is scheduled to run on the processor core with the lowest utilization in the appropriate scheduling domain. The data sharing advantage brought by the shared preset secondary cache can be fully utilized to reduce the delay of data access during task execution, thereby speeding up the execution speed of tasks, improving the execution efficiency of single tasks and the overall performance of the system for multi-tasking. The core selection scheduling of ready tasks is optimized to achieve refined scheduling at the CPU core granularity, maximizing the guarantee of the running performance of application tasks. Finally, with the operation of the embedded operating system, tasks are constantly generated and terminated, and the system environment and task load conditions are in dynamic changes. By updating the system environment information and task load information, these changes can be tracked in real time, so that subsequent task scheduling decisions can always be adjusted based on the latest and accurate system status.

[0103] Furthermore, an embodiment of the present invention also discloses an electronic device, Figure 4 It is a structural diagram of an electronic device according to an exemplary embodiment, and the content in the figure cannot be regarded as any limitation on the scope of use of the present invention. The electronic device may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input and output interface 25 and a communication bus 26. Among them, the memory 22 is used to store a computer program, and the computer program is loaded and executed by the processor 21 to implement the relevant steps in the task scheduling method based on reinforcement learning disclosed in any of the aforementioned embodiments. In addition, the electronic device in this embodiment may specifically be an electronic computer.

[0104] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device; the communication interface 24 can create a data transmission channel between the electronic device and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present invention, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0105] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0106] The operating system 221 is used to manage and control the hardware devices and computer program 222 on the electronic device, which can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program that can be used to complete the task scheduling method based on reinforcement learning performed by the electronic device disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to complete other specific tasks.

[0107] Furthermore, the present invention also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the aforementioned disclosed task scheduling method based on reinforcement learning is implemented. For the specific steps of the method, reference may be made to the corresponding contents disclosed in the aforementioned embodiments, and no further description will be given here.

[0108] Furthermore, the present invention also discloses a computer program product, including a computer program / instruction, which implements the steps of the method disclosed in any of the aforementioned embodiments when executed by a processor.

[0109] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0110] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0111] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0112] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0113] The technical solution provided by the present invention is introduced in detail above. Specific examples are used in this article to illustrate the principle and implementation mode of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation mode and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A task scheduling method based on reinforcement learning, characterized in that: include: Determine the task scheduling action of the current time step according to the current action experience value table and the preset scheduling constraints; The task scheduling action includes a first action, a second action and a third action determined based on the target experience value corresponding to the current time step in the current action experience value table; the first action is used to select a target processor node for executing the target task, the second action is used to select a target scheduling domain in the target processor node, and the third action is used to select a processor core in the target scheduling domain; each scheduling domain is configured with a number of processor cores that can share a preset secondary cache; Based on the current reward information after the embedded operating system executes the task scheduling action, updating the current system environment information and the current task load information of the embedded operating system; Based on the current system environment information, the current task load information and the current reward information, the experience value in the current action experience value table is updated, and the step of determining the task scheduling action of the current time step according to the current action experience value table and the preset scheduling constraints is jumped to, until the target task is scheduled to the target processor core that meets the preset conditions for task execution.

2. The task scheduling method based on reinforcement learning according to claim 1, characterized in that: Before updating the experience value in the current action experience value table based on the current system environment information, the current task load information and the current reward information, the method further includes: Obtain the current processor hardware architecture information and current task information of the embedded system to obtain the current system environment information; The current hardware utilization status information and the current task running status information of the embedded system are obtained to obtain the current task load information.

3. The task scheduling method based on reinforcement learning according to claim 2, characterized in that: Before determining the task scheduling action of the current time step according to the current action experience value table and the preset scheduling constraint conditions, the method further includes: The first affinity rule of the processor node, the second affinity rule of the scheduling domain, and the third affinity rule of the processor core in the processor hardware architecture information are set hierarchically based on the preset scheduling target; the preset scheduling target is a scheduling target determined based on having the best system performance in the scheduling process; A preset scheduling constraint condition for a scheduling task process is constructed based on the first affinity rule, the second affinity rule, and the third affinity rule.

4. The task scheduling method based on reinforcement learning according to claim 3 is characterized in that: The step of determining the task scheduling action of the current time step according to the current action experience value table and the preset scheduling constraint conditions includes: Determine a processor node selection action experience value corresponding to the current time step in the current action experience value table, so as to select a target processor node for executing the target task that satisfies the first affinity rule from local multi-processor nodes based on the processor node selection action experience value; Determine a scheduling domain selection action experience value corresponding to the current time step in the current action experience value table, so as to select a target scheduling domain that satisfies the second affinity rule from each of the scheduling domains in the target processor node based on the scheduling domain selection action experience value; The processor core selection action experience value corresponding to the current time step in the current action experience value table is determined, so as to select a processor core that satisfies the third affinity rule from each of the processor cores in the scheduling domain based on the processor core selection action experience value.

5. The task scheduling method based on reinforcement learning according to claim 3 is characterized in that: The first affinity rule of the processor node, the second affinity rule of the scheduling domain, and the third affinity rule of the processor core in the processor hardware architecture information are set hierarchically based on the preset scheduling target, including: A processor node utilization threshold in the processor hardware architecture information is set based on the preset scheduling target, so as to construct a first affinity rule for the task scheduling process according to a size relationship between a current processor node utilization and the processor node utilization threshold; A second affinity rule is set based on the preset scheduling target to schedule the execution task threads of the same task and the process tasks with data sharing to the same scheduling domain; A third affinity rule is set based on the preset scheduling target to schedule the task to the processor core with the lowest utilization rate in the scheduling domain.

6. The task scheduling method based on reinforcement learning according to claim 2, characterized in that: The obtaining of the current processor hardware architecture information and current task information of the embedded system includes: Acquire the associated information of the hardware information of the current system environment of the embedded operating system to obtain the current processor hardware architecture information constructed based on the physical structures of the processor and the interconnection information between the physical structures and the preset secondary cache, wherein the physical structure includes a processor node, a scheduling domain, and a processor core; The task priority, task cycle information, task execution thread, and data dependency relationship between tasks of the current task in the embedded operating system are obtained to obtain corresponding current task information.

7. The task scheduling method based on reinforcement learning according to claim 6, characterized in that: Before obtaining the current processor hardware architecture information and current task information of the embedded system, the method further includes: The processor cores on the embedded operating system are divided according to whether they share a preset secondary cache to obtain a scheduling domain; Dividing each of the scheduling domains according to whether the local memory of the same node is accessed to obtain corresponding processor nodes; The physical structure of the processor is determined according to the architectural relationship among the processor node, the scheduling domain, and the processor core.

8. The task scheduling method based on reinforcement learning according to claim 2, characterized in that: The obtaining of current hardware utilization status information of the embedded system includes: The utilization rate of the processor core and the usage of the shared preset secondary cache in the embedded operating system when the current system is running are obtained to obtain corresponding current hardware utilization status information.

9. The task scheduling method based on reinforcement learning according to claim 4, characterized in that: The determining of the processor node selection action experience value corresponding to the current time step in the current action experience value table, so as to select a target processor node for executing the target task that satisfies the first affinity rule from the local multi-processor nodes based on the processor node selection action experience value, comprises: Determine the maximum processor node selection action experience value in the current action experience value table, and select a processor node whose current processor node utilization is less than a processor node utilization threshold and has a task load from local multi-processor nodes based on the maximum processor node selection action experience value as the target processor node for executing the target task.

10. The task scheduling method based on reinforcement learning according to claim 9, characterized in that: The selecting, from the local multi-processor nodes, a processor node whose current processor node utilization is less than a processor node utilization threshold and has a task load and is determined as a target processor node for executing the target task, includes: Obtaining a current processor node utilization rate of each of the processor nodes recorded in the task load information; Selecting a processor node whose current processor node utilization is less than the processor node utilization threshold as an initial processor node; Determining whether there is a processor node with an existing task load among the initial processor nodes; If there is only one processor node with a task load among the initial processor nodes, the processor node with the task load is determined as the target processor node for executing the target task.

11. The task scheduling method based on reinforcement learning according to claim 10, characterized in that: After determining whether there is a processor node with an existing task load in the initial processor node, the method further includes: If there are multiple processor nodes with existing task loads among the initial processor nodes, the processor node with the existing task load that is closest in physical distance is determined as the target processor node for executing the target task.

12. The task scheduling method based on reinforcement learning according to any one of claims 1 to 11, characterized in that: Also includes: Based on the values ​​of each state parameter in the state parameter variation range generated by each scheduling action, determine the task scheduling action that can be executed at the current time step to construct a target action set; the state parameter variation range is the state parameter variation range corresponding to the current system environment information and the current task load information at the current time step; Acquire the experience value corresponding to each action in the target action set from the current action experience value table to obtain the experience value set corresponding to the current time step; The target experience value corresponding to the current time step is determined from the experience value set according to the highest value selection rule.

13. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to execute the computer program to implement the steps of the task scheduling method based on reinforcement learning as described in any one of claims 1 to 12.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the task scheduling method based on reinforcement learning as described in any one of claims 1 to 12 are implemented.

15. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps of the task scheduling method based on reinforcement learning described in any one of claims 1 to 12 are implemented.

Images