Method, apparatus, device, storage medium and program product for resource scheduling

Abstract

A method, device, equipment, storage medium and program product for resource scheduling are provided. The method comprises: determining a task type of a reinforcement learning task of a machine learning model; based on the task type, determining resource requirement information corresponding to the task type, the resource requirement information being used to describe a required computing resource of the reinforcement learning task; based on the resource requirement information, determining a scheduling scheme, the scheduling scheme being used to indicate a computing resource used to execute the reinforcement learning task; and based on the scheduling scheme, scheduling the computing resource to execute the reinforcement learning task. In this way, differentiated and dynamic computing resource scheduling can be implemented for the type characteristics and stage characteristics of the reinforcement learning task, and the utilization rate of the heterogeneous computing resource and the reinforcement learning training efficiency are improved.

Description

Technical Field

[0001] The exemplary embodiments disclosed herein generally relate to the field of information technology, and particularly to methods, apparatus, devices, computer-readable storage media, and computer program products for resource scheduling. Background Technology

[0002] With the development of artificial intelligence technology, especially the increasing popularity of large-scale model training based on reinforcement learning (RL) (such as embodied intelligence, intelligent agent tools, inference models, etc.), new challenges have been posed to the scheduling and management of distributed computing resources. Summary of the Invention

[0003] In a first aspect of this disclosure, a method for resource scheduling is provided. The method includes: determining a task type for a reinforcement learning task of a machine learning model; determining resource requirement information corresponding to the task type, the resource requirement information describing the computational resources required for the reinforcement learning task; determining a scheduling scheme based on the resource requirement information, the scheduling scheme indicating the computational resources used to execute the reinforcement learning task; and scheduling the computational resources based on the scheduling scheme to execute the reinforcement learning task.

[0004] In a second aspect of this disclosure, an apparatus for resource scheduling is provided. The apparatus includes: a task type determination module configured to determine a task type of a reinforcement learning task of a machine learning model; a resource requirement determination module configured to determine resource requirement information corresponding to the task type based on the task type, the resource requirement information describing the resource requirements of the reinforcement learning task; a scheduling scheme determination module configured to determine a scheduling scheme based on the resource requirement information, the scheduling scheme indicating computational resources for executing the reinforcement learning task; and a scheduling module configured to schedule computational resources based on the scheduling scheme to execute the reinforcement learning task.

[0005] In a third aspect of this disclosure, an electronic device is provided. The electronic device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method of the first aspect of this disclosure when executed by the at least one processing unit.

[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. This computer-readable storage medium stores a computer program that can be executed by a processor to perform the method according to a first aspect of this disclosure.

[0007] In a fifth aspect of this disclosure, a computer program product is provided, which is tangibly stored in a computer storage medium and includes computer-executable instructions that, when executed by a device, cause the device to perform the method according to a first aspect of this disclosure.

[0008] This approach enables differentiated and dynamic resource scheduling tailored to the characteristics of reinforcement learning task types, significantly improving the utilization rate of heterogeneous computing resources, enhancing the overall efficiency of reinforcement learning training, and reducing training costs.

[0009] It should be understood that the content described in this section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0010] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein: Figure 1 A schematic diagram of the example environment is shown; Figure 2 Flowcharts of example processes for resource scheduling in several scenarios are shown; Figure 3 Block diagrams of example architectures for scheduling engines in some scenarios are shown; Figure 4 The diagram illustrates example architectures for resource scheduling in several scenarios; Figure 5 Block diagrams of systems for processing tasks in several scenarios are shown; and Figure 6 A block diagram of an electronic device capable of implementing multiple illustrative scenarios is shown. Detailed Implementation

[0011] The examples in the text will now be described in more detail with reference to the accompanying drawings. While some examples are shown in the drawings, it should be understood that solutions can be implemented in various forms and should not be construed as limited to the examples presented herein. Rather, these examples are provided to provide a more thorough and complete understanding of the solutions. It should be understood that the drawings and examples in this document are for illustrative purposes only and are not intended to limit the scope of protection of the solutions.

[0012] It should be noted that the headings of any section / subsection provided herein are not restrictive. Various examples are described throughout this document, and examples of any type may be included under any section / subsection. Furthermore, examples described in any section / subsection may be combined in any way with any other examples described in the same section / subsection and / or different sections / subsections.

[0013] In the description of the examples in this document, the term "including" and similar terms should be understood as open inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "an example" or "the example" should be understood as "at least one example". The term "some examples" should be understood as "at least some examples". Other explicit and implicit definitions may also be included below. The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0014] The examples in this document may involve user data, data acquisition, and / or use. All of these aspects comply with relevant laws, regulations, and rules. In the examples, all data collection, acquisition, processing, manipulation, forwarding, and use are conducted with the user's knowledge and confirmation. Accordingly, when implementing each example, the type, scope of use, and usage scenarios of any data or information that may be involved should be communicated to the user and their authorization obtained through appropriate means, in accordance with relevant laws and regulations. The specific methods of notification and / or authorization can vary depending on the actual situation and application scenario; the scope of the solution is not limited in this regard.

[0015] In this manual and the sample solutions, any processing of personal information will be conducted only under legal grounds (such as obtaining the consent of the data subject or being necessary for the performance of a contract) and will only be carried out within the scope stipulated or agreed upon. A user's refusal to process personal information beyond what is necessary for basic functions will not affect the user's use of basic functions.

[0016] The term "model" as used in this paper refers to a model that learns the relationship between inputs and outputs from training data, enabling it to generate corresponding outputs for a given input after training. Model generation can be based on machine learning techniques. Deep learning is a machine learning algorithm that uses multiple layers of processing units to process inputs and provide corresponding outputs. A neural network model is an example of a deep learning-based model. In this paper, "model" may also be referred to as a "machine learning model," "learning model," "machine learning network," or "learning network," and these terms are used interchangeably. The term "reinforcement learning task" as used in this paper refers to the task of training a machine learning model using reinforcement learning, which may include, but is not limited to, model training tasks for scenarios such as embodied intelligence, agent tool invocation, and logical reasoning.

[0017] As mentioned above, with the development of artificial intelligence technology, especially the increasing popularity of large-scale model training based on reinforcement learning (RL) (such as embodied intelligence, intelligent agent tools, inference models, etc.), unprecedented challenges have been posed to the scheduling and management of distributed computing resources.

[0018] In some scenarios, a scheduler can be used for container scheduling based on a two-stage strategy of pre-selection and optimization. This scheduler allocates containers to suitable nodes through mechanisms such as resource request volume, node affinity, anti-affinity rules, and tolerance. However, its scheduling decisions are mainly geared towards stateless services or general batch processing tasks, lacking specialized awareness of reinforcement learning training tasks with complex stage characteristics and differentiated resource requirements.

[0019] In other scenarios, a master-slave architecture can be used for computing resource scheduling. This architecture supports both fair schedulers and capacity schedulers. Master-slave architectures are primarily used in big data batch processing scenarios. Their scheduling granularity is relatively coarse, using static resource containers as the basic allocation unit, and they lack the ability to perceive and respond to dynamic changes during task execution.

[0020] In some cases, workload schedulers can be used for computational resource scheduling. Workload schedulers support various scheduling strategies, including job submission, resource allocation, parallel task execution, and queue management. However, in deep learning training scenarios, their static resource reservation and exclusive allocation modes result in low resource utilization and do not support dynamic resource adjustments during training.

[0021] Therefore, traditional schedulers suffer from the following main problems when facing reinforcement learning training scenarios: First, traditional schedulers cannot accurately match the resource requirements of different reinforcement learning tasks, resulting in low matching between tasks and hardware and impaired training efficiency. Second, reinforcement learning training involves multiple typical stages, each with significantly different resource requirements. Traditional schedulers use static resource allocation and cannot dynamically adjust resource configuration according to the current stage, leading to resource waste in stages that do not require large amounts of computing resources, and resource shortages in stages that require large amounts of computing resources. Third, traditional schedulers lack a cross-stage elastic resource transfer mechanism. This resource allocation mode results in long task queuing times, low overall cluster throughput, and high training costs. Therefore, traditional scheduling strategies mainly rely on static resource allocation strategies for general computing tasks, lacking the ability to perceive the type and stage characteristics of reinforcement learning training tasks, resulting in low matching between resources and tasks and low resource utilization.

[0022] In view of this, this paper proposes a resource scheduling scheme. According to the scheme, firstly, the task type of the reinforcement learning task of the machine learning model is determined; then, the resource requirement information corresponding to the task type is determined based on the task type; next, a scheduling scheme for executing the reinforcement learning task is determined based on the resource requirement information; and finally, computing resources are scheduled to execute the reinforcement learning task based on the scheduling scheme.

[0023] This approach enables differentiated and dynamic resource scheduling tailored to the characteristics of reinforcement learning task types, significantly improving the utilization of heterogeneous computing resources, shortening task queuing time, enhancing the overall efficiency of reinforcement learning training, and reducing training costs.

[0024] The following describes various examples of this scheme in further detail with reference to the accompanying drawings.

[0025] Figure 1 A schematic diagram of example environment 100 is shown. (e.g.) Figure 1 As shown, example environment 100 may include, for example, a system for performing reinforcement learning training tasks (e.g., it may be referred to as a training system). In some examples, the training system may adopt a layered and decoupled architecture, which may include, from top to bottom, an access layer 110, a scheduling layer 120, an adaptation layer 130, an engine layer 140, a data layer 150, and a computing resource pool 160.

[0026] Access layer 110 is a unified task access and representation layer, configured to receive and represent reinforcement learning tasks, and is located at the top layer of the architecture. In some examples, access layer 110 can receive task requests for reinforcement learning tasks and parse the task requests into standardized task descriptions. The task descriptions may include, but are not limited to, at least one of the following: task type identifier, model configuration, environment configuration, resource requirement description, and recommended hyperparameters.

[0027] The scheduling layer 120 is configured to orchestrate and schedule reinforcement learning tasks. In some examples, the scheduling layer 120 can determine the task execution plan (e.g., a directed acyclic graph) of the reinforcement learning task based on the task description provided by the access layer 110. The scheduling layer 120 can also be used to perform full lifecycle management of reinforcement learning tasks. In some examples, the scheduling layer 120 can determine the task type and at least one task stage of the reinforcement learning task based on the task execution plan, thereby determining a scheduling scheme indicating computing resources, and scheduling computing resources in the computing resource pool 160 based on the scheduling scheme. In some examples, such as... Figure 1 As shown, the scheduling layer 120 may include a scheduling engine 125, which can be used to determine a scheduling scheme for reinforcement learning tasks.

[0028] The adaptation layer 130 is configured to provide a unified environment access interface for different types of reinforcement learning tasks, adapting to the underlying differences between different environments such as physical simulation environments, tool calling environments, and logical reasoning verification environments. The engine layer 140 is configured to provide the computational capabilities required for reinforcement learning tasks, such as training, inference, and environment execution. The data layer 150 is configured to provide data services required for reinforcement learning tasks, such as sample and trajectory management, reward calculation and aggregation, and model parameter synchronization.

[0029] Computational resource pool 160 is configured to provide computing resources. In some examples, such as... Figure 1 As shown, the computing resource pool 160 may include a first resource cluster, a second resource cluster, a third resource cluster, and a fourth resource cluster. The first resource cluster may, for example, include a training cluster composed of graphics processing units (GPUs); the second resource cluster may, for example, include an environment cluster composed of GPUs; the third resource cluster may, for example, include an inference cluster composed of GPUs; and the fourth resource cluster may, for example, include a sandbox cluster composed of central processing units (CPUs). It should be understood that the types and numbers of the resource clusters described above are merely exemplary and can be flexibly configured according to actual needs.

[0030] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of the scheme. The example will continue to be described below with reference to the accompanying drawings.

[0031] Figure 2A flowchart of an example process 200 for processing a task is shown, depending on several scenarios. Process 200 can be implemented at the training system. In some cases, process 200 can be implemented at the scheduling layer 120 (e.g., at the scheduling engine 125) of the training system. For ease of description, process 200 will be described below from the perspective of the training system.

[0032] like Figure 2 As shown in box 210, the training system determines the task type of the reinforcement learning task for the machine learning model. A reinforcement learning task refers to the task of training the machine learning model using reinforcement learning methods; the task type can be determined based on the functional characteristics of the machine learning model.

[0033] In box 220, the training system determines the resource requirements corresponding to the task type. These resource requirements describe the computational resources needed for the reinforcement learning task.

[0034] In box 230, the training system determines a scheduling scheme based on resource requirement information, which indicates the computing resources used to perform reinforcement learning tasks.

[0035] In box 240, the training system schedules computing resources based on a scheduling scheme to execute reinforcement learning tasks. The training system can convert the scheduling scheme into scheduling instructions executable by the underlying orchestration system through orchestration adapter 324, thereby actually executing the reinforcement learning tasks on the target computing nodes in the computing resource pool 160.

[0036] Through the above process 200, the training system can dynamically determine the resource requirements of each reinforcement learning task based on its type characteristics, and generate corresponding scheduling schemes accordingly. This achieves automated processing from task type awareness to resource scheduling decisions, improving the matching accuracy between scheduling decisions and actual task requirements. The following, combined with... Figure 3 and Figure 4 The internal structure of the training system and the specific implementation methods of each step are explained in detail.

[0037] Figure 3 A block diagram of an example structure of the scheduling engine 125 is shown for some scenarios. Figure 4 The diagram illustrates example data flow 400 for resource scheduling in several scenarios. Specifically, Figure 4 It shows Figure 3 A schematic diagram of example data flow 400 between the various components of the scheduling engine 125. (See diagram below.) Figure 3 and Figure 4As shown, the scheduling engine 125 may include a task type identifier 302, also known as a Task Type Feature Profiler (TTFP). The task type identifier 302 may receive task execution plans, such as directed acyclic graphs, from the scheduling layer 120. Based on, for example, the task execution plan, the task type identifier 302 can determine the task type and, based on the task type, determine resource requirement information.

[0038] In some cases, the task type may include at least one of the following: a first task type, a second task type, and a third task type. The reinforcement learning task of the first task type is used to train a first machine learning model, which is configured to generate action instructions based on observations of a first environment. These action instructions are used to control objects in the first environment to perform actions. As an example, the first task type may correspond to an embodied intelligence scenario, such as a reinforcement learning task used to train a robot control model with physical entities capable of interacting with a physical environment. A typical first machine learning model may include, for example, a visual-language-action model.

[0039] The second task type of reinforcement learning task is used to train a second machine learning model, which is configured to generate invocation instructions for a tool. These instructions are used to invoke the tool via a second environment to perform operations corresponding to the invocation instructions. As an example, the second task type can correspond to an agent tool invocation scenario, such as a reinforcement learning task used to train an agent model capable of autonomously invoking external tools and performing multi-step tasks.

[0040] The third task type of reinforcement learning task is used to train a third machine learning model, which is configured to generate reasoning processes for logical reasoning tasks. As an example, the third task type can correspond to reasoning model scenarios, such as reinforcement learning tasks used to train large language models with deep reasoning capabilities.

[0041] In some scenarios, the task type recognizer 202 can determine resource requirement information based on the task type. A reinforcement learning task may include multiple task stages, and the resource requirement information may include at least one of time information, feature information, preference information, and policy information. Time information may indicate the duration of each of the multiple task stages. Feature information indicates the computational resources required by each of the multiple task stages. Preference information indicates the type of computational resources preferentially allocated to the multiple task stages (e.g., preferential allocation of computational resources with strong graphics rendering capabilities, or preferential allocation of computational resources with large video memory capacity). Policy information indicates the allocation strategy used to allocate computational resources to the reinforcement learning task (e.g., dedicated allocation strategy, shared allocation strategy, or stage-level flexible allocation strategy, etc.).

[0042] In some examples, the training system can pre-build resource requirement profiles (i.e., resource requirement information) corresponding to each task type. The resource requirement profile can include multiple profile dimensions, which may include, but are not limited to, typical model type, GPU computing intensity, memory requirement characteristics, CPU requirement, graphics rendering requirement, network bandwidth requirement, recommended GPU type, recommended allocation strategy, stage duration distribution characteristics, and special hardware preferences, etc.

[0043] As an example, for the first task type, the resource requirement profile can indicate: medium to high intensity computing power requirements (for physics simulation and policy training), medium video memory requirements (e.g., 8GB to 24GB), high CPU core requirements (for physics engine simulation), high graphics rendering requirements (for 3D or 2D environment rendering), medium network bandwidth requirements (for parameter synchronization), and a stage duration distribution characteristic with a high proportion of environment stages (e.g., 40% to 60%).

[0044] For the second task type, the resource requirement profile can indicate: high computing power requirements (for long context processing), high video memory requirements (e.g., 24GB or more), medium CPU core requirements (for tool call management), low graphics rendering requirements, low network bandwidth requirements (but sensitive to latency), and a stage duration distribution characteristic with a high proportion of inference stage (e.g., 50% to 70%).

[0045] For the third task type (inference model), the resource requirement profile can indicate: extremely high computing power requirements (for large-scale distributed inference), extremely high GPU memory requirements (e.g., more than 40GB), low CPU core requirements (purely computationally intensive), no graphics rendering requirements, extremely high network bandwidth requirements (for tensor parallel communication), and a stage duration distribution characteristic with a high proportion of training phase (e.g., 60% to 80%).

[0046] In some scenarios, the task type identifier 302 can automatically generate a resource profile for a task by analyzing historical task execution data (such as time-series data on resource utilization and stage duration distribution data). This method automates the resource profile generation process, but typically requires accumulating a certain amount of historical data. Alternatively and / or additionally, the task type identifier 302 can also determine the resource profile for a task based on the task type and resource requirement preference information manually specified by the user when submitting the reinforcement learning task. This method is simpler to implement, but relies on the user's accurate judgment of their own task.

[0047] Through the above-mentioned task type identification and resource requirement determination process, the training system can automatically establish differentiated resource requirement profiles for different types of reinforcement learning tasks such as embodied intelligence, intelligent agent tools, and inference models. This provides basic data support for subsequent differentiated scheduling and avoids the problem of low task-hardware matching caused by the "one-size-fits-all" resource allocation strategy of existing schedulers.

[0048] In some scenarios, the training system (e.g., the task stage perceptron 304) can acquire the identification results of the task stage in which the reinforcement learning task is located. Then, based on resource requirement information and the identification results, the training system can determine a scheduling scheme. In this way, scheduling decisions are not only based on task type but also further delineated to the task stage dimension. Because the resource requirement characteristics of each stage are incorporated into the scheduling considerations, the scheduling engine can determine the optimal resource configuration for different stages, avoiding resource waste caused by using the same resource configuration throughout the entire lifecycle, thus improving the time-dimensional utilization of cluster computing resources.

[0049] In some scenarios, the multiple task phases of a reinforcement learning task may include at least one of the following: an environment phase, an inference phase, a training phase, a reward determination phase, a parameter synchronization phase, and an evaluation phase. The environment phase is configured to initialize an environment instance corresponding to the reinforcement learning task, such as state progression or observation generation in a physical simulation environment or a virtual interactive environment. The inference phase is configured to use a machine learning model to determine an operation instruction for the environment instance based on first observation information, and to execute the operation within the environment instance using the operation instruction to obtain second observation information of the environment instance. The first and second observation information indicate the environmental state of the environment instance before and after the operation, respectively. The training phase is configured to perform reinforcement learning training on the machine learning model using training samples to obtain a trained machine learning model. The training samples are constructed based on at least one of the operation instruction, the first observation information, and the second observation information. The reward determination phase is configured to determine a reward value for the operation instruction based at least on the second observation information. The reward value indicates the correctness of the operation instruction. The parameter synchronization phase is configured to transfer the parameters of the trained machine learning model to at least one computing node, such as gradient aggregation and parameter synchronization in a distributed training scenario. The evaluation phase is configured to perform a task quality assessment on the reinforcement learning task to obtain evaluation results, such as evaluating the performance of the current policy on a validation environment. It should be understood that the names and divisions of the six task phases described above are merely illustrative. In some cases, multiple task phases may include more, fewer, or different phases; this document does not impose any limitations on this.

[0050] The training system can acquire feature information and multiple reference information. The feature information indicates the running state of the reinforcement learning task, and the multiple reference information indicates multiple task stages of the reinforcement learning task. In some cases, the training system can acquire multiple primary indicators, which indicate the usage status of the reinforcement learning task on multiple types of computing resources. Then, the training system can perform feature extraction based on these primary indicators to obtain feature information. As an example, the task stage perceptron 304 can acquire multiple primary indicators, which indicate the usage status of the reinforcement learning task on multiple types of computing resources, such as, but not limited to, GPU utilization, memory usage, CPU utilization, network bandwidth usage, ensemble communication activity, I / O read / write speed, and other multi-dimensional indicators. Then, feature extraction (e.g., feature concatenation, normalization, encoding, etc.) is performed on these primary indicators to obtain comprehensive feature information, which constitutes a comprehensive feature vector representing the current running state of the reinforcement learning task.

[0051] The training system can determine the matching degree between feature information and multiple reference information to obtain multiple first scores. The training system can determine whether the multiple first scores (also called matching scores) exceed a first threshold. If one of the multiple first scores exceeds the first threshold, it is determined that the reinforcement learning task is in the first task stage corresponding to the first score that exceeds the first threshold, or it is determined that the reinforcement learning task will switch to the first task stage.

[0052] The reference information here can be used to describe the corresponding task phase. In some cases, the reference information may include one or more reference metrics that indicate the utilization of one or more types of computing resources by the reinforcement learning task at the corresponding task phase. For example, multiple reference metrics corresponding to the environment phase may indicate high CPU utilization, moderate GPU utilization (e.g., for environment rendering), increased I / O read / write operations, etc. Reference metrics corresponding to the inference phase may indicate high GPU utilization (e.g., exceeding 70%), moderate memory usage, and the generation of a large amount of empirical data, etc. Reference metrics corresponding to the training phase may indicate high GPU utilization (e.g., exceeding 80%), high GPU memory usage, and a high proportion of computationally intensive instructions, etc. The reward determination phase may involve the computation of complex reward models, and its reference metrics may indicate mixed GPU and CPU loads, short bursts of computation, and moderate communication volume, etc. Multiple reference metrics corresponding to the parameter synchronization phase may indicate high network bandwidth usage, low GPU computational load, and active ensemble communication, etc. Several reference metrics corresponding to the evaluation phase can indicate that the evaluation phase has moderate GPU utilization (e.g., 50% to 70%), stable load, no backpropagation computation, etc.

[0053] As an example, the task stage perceptron 304 acquires multiple reference information items, each indicating a different task stage of the reinforcement learning task. For example, each task stage corresponds to a preset threshold template. The task stage perceptron 304 determines the matching degree between feature information and the multiple reference information items to obtain multiple first scores. Each first score indicates the degree of matching between the feature information and the reference information of the corresponding task stage. When the first score corresponding to a certain task stage exceeds a preset confidence threshold (i.e., the first threshold) for that task stage, that task stage enters the candidate state. In response to one of the multiple first scores exceeding the first threshold, the task stage perceptron 304 determines that the reinforcement learning task is in the first task stage or is about to switch to the first task stage. The first task stage is the task stage among the multiple task stages that corresponds to the first score exceeding the first threshold.

[0054] In some cases, different task stages can correspond to different first thresholds. For example, the first threshold for the training and inference stages can be set to 0.85, the first threshold for the environment and parameter synchronization stages can be set to 0.80, and the first threshold for the reward determination and evaluation stages can be set to 0.75, to reflect the differences in recognition difficulty and misjudgment cost at different stages. Of course, the above thresholds are only examples, and any other appropriate value can be selected according to actual needs. This article does not impose any restrictions on this.

[0055] In some scenarios, the training system can determine that the reinforcement learning task is in or about to switch to the first task stage based on the fact that the first score corresponding to the first task stage exceeds a first threshold in multiple consecutive recognition cycles. To avoid misjudgments due to instantaneous fluctuations, the task stage perceptron 304 introduces a lag confirmation window mechanism. Specifically, the task stage perceptron 304 only formally determines that the reinforcement learning task is in or about to switch to the first task stage based on the fact that the first score corresponding to the first task stage exceeds a first threshold in multiple consecutive recognition cycles. Different task stages can correspond to different confirmation window lengths. For example, the confirmation window for the training and inference stages can be 3 to 5 recognition cycles, the confirmation window for the environment and parameter synchronization stages can be 2 to 4 recognition cycles, and the confirmation window for the reward determination and evaluation stages can be 2 to 3 recognition cycles. If other candidate task stages with higher matching degrees appear during the lag confirmation window, the counting restarts.

[0056] Through the triple verification mechanism combining multi-indicator fusion, threshold judgment, and lag confirmation window, the task phase perceptron 304 can effectively avoid phase misjudgment caused by instantaneous fluctuations of indicators while ensuring recognition accuracy, thereby providing accurate and stable task phase recognition results for subsequent scheduling decisions.

[0057] In some cases, the task phase perceptron 304 can also determine the Phase Parallelism Index (PPI) to quantify the parallel expansion potential of the current task phase. The Phase Parallelism Index can be determined as follows: (1) in, This represents the current actual amount of resources allocated. This represents the optimal resource requirement for this phase of the task. This represents the percentage of time that can be executed in parallel during this task phase. This represents the total duration of this task phase. A value closer to 1 indicates a more efficient allocation of computational resources for the current task phase.

[0058] In some cases, the task phase perceptron 304 can also employ, for example, an exponentially weighted moving average (EWMA) model to predict the duration of a task phase. The predicted phase duration can be determined as follows: (2) in, The actual duration of the task phase observed in the nth instance. Let be the predicted duration of the nth task phase. The smoothing factor (e.g., 0.3 to 0.5) can be used as a reference for the scheduling decision engine 312 and the dynamic scaling controller 322 to achieve predictive resource adjustments.

[0059] In some scenarios, the training system can determine multiple second scores corresponding to multiple candidate computing resources, each indicating the data processing capacity of the candidate computing resources. Based on these second scores, resource requirement information, and the identification results of the task stage, the training system can determine the matching degree between the multiple candidate computing resources and the reinforcement learning task, thus obtaining multiple third scores. Subsequently, based on these third scores, the training system can select computing resources from the multiple candidate computing resources to perform the reinforcement learning task, thereby obtaining a scheduling scheme.

[0060] like Figure 3 and Figure 4As shown, a second score for candidate computing resources (e.g., candidate computing nodes) can be determined using a heterogeneous hardware-aware scheduler 314. The heterogeneous hardware-aware scheduler 314 is responsible for evaluating and comprehensively scoring the heterogeneous computing resources (e.g., different models of GPUs, NPUs, etc.) in the computing resource pool 160. The second score is determined based on at least two of the following: the computing power of the candidate computing resource; the video memory capacity of the candidate computing resource; the interconnect bandwidth of the candidate computing resource; the graphics rendering capability of the candidate computing resource; and the cost-effectiveness of the candidate computing resource.

[0061] As an example, the heterogeneous hardware-aware scheduler 314 can adopt a five-dimensional hardware capability scoring model. The five scoring dimensions and their example weights can be: computing power (e.g., the weight can be 35% or other values, and the evaluation indicators can include FP16 computing power, INT8 computing power, etc.), video memory capacity (e.g., the weight can be 25% or other values, and the evaluation indicators can include video memory capacity size, video memory bandwidth, etc.), interconnect bandwidth (e.g., the weight can be 20% or other values, and the evaluation indicators can include multi-card interconnect bandwidth, multi-card expansion efficiency, etc.), graphics rendering (e.g., the weight can be 15% or other values, and the evaluation indicators can include graphics processing performance, physics simulation acceleration capability, etc.), and cost-effectiveness (e.g., the weight can be 5% or other values, and the evaluation indicator can be the ratio of hardware performance to procurement cost).

[0062] The heterogeneous hardware-aware scheduler 314 can match hardware with tasks based on the "specific hardware preference" dimension in the task's resource requirement profile, combined with the comprehensive score of each candidate computing resource. For example, for tasks of the first task type (embodied intelligence), computing resources with higher scores in the graphics rendering dimension are prioritized; for tasks of the second task type (agent tools), computing resources with higher scores in the video memory capacity dimension are prioritized; and for tasks of the third task type (inference models), computing resources with higher scores in the computing power and interconnect bandwidth dimensions are prioritized. The second score output by the heterogeneous hardware-aware scheduler 314 provides a quantitative basis for the scheduling decision engine 312 to determine the third score.

[0063] In some scenarios, the scheduling decision engine 312 can determine scores across multiple dimensions based on resource requirement information and task stage identification results. These scores may include a second score. Subsequently, the scheduling decision engine 312 can determine a third score based on the scores across these multiple dimensions and their respective weights. For example, these multiple dimensions may include a resource requirement profile matching dimension, an affinity dimension, a load balancing and fragmentation defragmentation dimension, a fault tolerance and failover dimension, a task type profile matching dimension, a heterogeneous hardware scoring dimension (i.e., the second score), and a stage parallelism optimization dimension. The resource requirement profile matching dimension is used to match the optimal combination of node resources based on the resource profile. The affinity matching dimension considers data locality and inter-task communication patterns, scheduling related tasks to neighboring nodes. The load balancing and fragmentation defragmentation dimension balances the load on each node and reduces resource fragmentation. The fault tolerance and failover dimension considers node health status and historical failure rates. The task type profile matching dimension matches tasks based on specific task type preferences. The heterogeneous hardware scoring matching dimension combines the second score output by the heterogeneous hardware-aware scheduler 314 to select the most suitable computing resources. The stage parallelism optimization dimension is used to determine whether additional parallel resources need to be allocated to the current task stage based on the stage parallelism index.

[0064] As an example, the aforementioned dimensions can be assigned different weights. For instance, the weights of the resource requirement profile matching dimension, affinity matching dimension, load balancing and defragmentation dimension, fault tolerance and failover dimension, task type profile matching dimension, heterogeneous hardware score matching dimension, and stage parallelism optimization dimension can be set to approximately 25%, 15%, 15%, 10%, 15%, 15%, and 5%, respectively. The scheduling decision engine 312 calculates a comprehensive scheduling score (i.e., the third score) for the candidate computing resources and selects the candidate computing resource with the highest third score as the target computing resource. When multiple dimensions conflict (e.g., the candidate computing resource with a high resource requirement profile matching degree has a high load), the scheduling decision engine 312 can use a weighted scoring method to make a trade-off decision. It should be understood that the number, name, and weight allocation of the above optimization dimensions are only examples. In other embodiments, more, fewer, or different combinations of optimization dimensions can be used, and this document does not impose any restrictions on this.

[0065] Through the above-mentioned scheduling scheme determination process based on heterogeneous hardware scoring and multi-dimensional optimization, the scheduling decision engine 312 can comprehensively consider multiple factors such as task requirements, task stage and hardware capabilities to achieve accurate matching between candidate computing resources and reinforcement learning tasks. This overcomes the problem of insufficient matching accuracy caused by existing schedulers relying only on single or a few indicators such as resource request volume for scheduling.

[0066] In some scenarios, the heterogeneous hardware-aware scheduler 314 can operate independently of preset scoring weights, instead periodically running standardized benchmark programs within the cluster and dynamically updating the scores of each computing resource based on actual test results. This approach yields scores that more closely reflect actual hardware performance, but the benchmark tests themselves consume certain cluster resources. Alternatively and / or additionally, the heterogeneous hardware-aware scheduler 314 can also support users customizing the weighting of the five scoring dimensions according to their task characteristics. This approach offers greater flexibility but requires users to have a deep understanding of the resource requirements of their tasks.

[0067] In some cases, such as Figure 3 and Figure 4 As shown, the dynamic scaling controller 322 is responsible for dynamically adjusting resource allocation according to changes in task load. Its adjustment methods can include reactive elastic scaling and predictive elastic scaling. In reactive elastic scaling, the dynamic scaling controller 322 monitors resource utilization metrics of the reinforcement learning task in real time (e.g., GPU utilization, memory usage, CPU utilization, etc.). If a metric consistently exceeds a preset upper threshold (e.g., GPU utilization exceeds 85% for 30 seconds), the dynamic scaling controller 322 triggers a scaling-up operation. If a metric consistently falls below a preset lower threshold (e.g., GPU utilization is below 30% for 60 seconds), the dynamic scaling controller 322 triggers a scaling-down operation.

[0068] In some scenarios, if the task phase identification result indicates that the reinforcement learning task is about to switch from the second task phase to the third task phase, the dynamic scaling controller 322 pre-initializes the computing resources required for the third task phase on the first computing node based on the scheduling scheme. If it is determined that the second task phase has ended, the third task phase is executed on the first computing node. As an example, the dynamic scaling controller 322 has a built-in predictive scaling engine that prepares resources in advance based on the phase switching prediction result output by the task phase perceptron 304 and the phase duration prediction result described above. Specifically, the predictive scaling engine can pre-allocate and initialize the computing resources required for the third task phase on the target computing node (i.e., the first computing node) in advance when it is predicted that a phase switch is about to occur (e.g., 30 seconds to 2 minutes before the predicted switch time). The predictive scaling engine can also combine the phase parallelism index to determine the trend of resource demand changes in the next task phase after the current task phase ends. Predictive elastic scaling can reduce the response time of expansion from 10 to 30 seconds for reactive elastic scaling to near 0 seconds (because the required resources are already prepared in advance), thereby effectively avoiding resource contention and performance jitter during training.

[0069] In some scenarios, the dynamic scaling controller 322 receives the task stage identification result and stage parallelism index output by the task stage perceptron 304. It queries the current resource allocation status determined by the scheduling decision engine 312. It evaluates the difference between the currently allocated resource amount and the optimal resource amount for the target task stage. If predictive elastic scaling is triggered, computing resources are allocated 30 seconds to 2 minutes in advance. If reactive elastic scaling is triggered, resource additions or releases of excess resources are completed within 10 seconds. Afterwards, the dynamic scaling controller 322 can update the resource allocation status and notify the orchestration adapter 324 to perform resource scaling operations.

[0070] In some scenarios, the training system can migrate task data from a second computing node to a third computing node in response to a transfer condition triggered by the reinforcement learning task. This task data indicates the task state of the reinforcement learning task. Subsequently, the training system can execute the fifth task phase of the reinforcement learning task on the third computing node according to this task state.

[0071] The migration conditions include at least the following: the reinforcement learning task is about to switch from the fourth task stage to the fifth task stage, the second computing node is used to execute the fourth task stage, and the third computing node is used to execute the fifth task stage. In some cases, the migration conditions may also include at least one of the following: the task stage perceptron 304 detects the task stage switch, and the optimal hardware configuration of the target task stage does not match the hardware of the current computing node; the stage parallelism index determined by the dynamic scaling controller 322 shows that the difference between the current resource configuration and the resource requirements of the target task stage is greater than the response threshold (e.g., the stage parallelism index is less than 0.5); and the scheduling decision engine 312 finds that other computing nodes can provide a better third score for the next task stage.

[0072] In some cases, the cross-stage hot migration service 332 assesses the necessity and feasibility of the migration. Specifically, the cross-stage migration service 332 may assess: the expected benefits of the migration (e.g., the difference between the overall scheduling score of the third compute node and the overall scheduling score of the second compute node) and the estimated migration costs (e.g., migration time, network bandwidth consumption, risk of interruption to the training process, etc.). If the expected benefits outweigh the migration costs, the migration is approved; otherwise, the migration is canceled.

[0073] The cross-stage hot migration service 332 determines the amount of task data to be migrated. If it is the first migration (i.e., full migration), it determines the total amount of all task data (i.e., full data), including complete model parameters, optimizer state, training context, etc. If it is a subsequent migration (i.e., incremental migration), it determines the amount of task data (i.e., incremental data) that has changed since the last migration through a difference comparison algorithm.

[0074] Before the actual transfer of task data, the cross-stage hot migration service 332 can prepare the execution environment on the third compute node in advance. Specifically, this may include starting the third compute node, pre-allocating computing resources (such as GPU and video memory resources) on the third compute node, loading the basic model weights (which can be reused if the third compute node already has the corresponding cache), and establishing a data transfer channel between the second and third compute nodes, etc.

[0075] After the data transmission channel is established, the cross-stage hot migration service 332 can migrate the task data of the reinforcement learning task from the second computing node to the third computing node. In some cases, the cross-stage migration service 322 can determine whether the migration is a full migration or an incremental migration. If the migration is a full migration, the cross-stage migration service 332 can transfer all task data from the second computing node to the third computing node. If the migration is an incremental migration, the cross-stage migration service 332 can identify the incremental data in the task data, which is the data that has changed since the previous migration. The cross-stage migration service 332 can then transfer the incremental data from the second computing node to the third computing node.

[0076] In some scenarios, the cross-stage hot migration service 332 can leverage high-speed transmission technologies such as Remote Direct Memory Access (RDMA) to improve bandwidth utilization. During transmission, the second compute node can continue executing reinforcement learning tasks and employ a copy-on-write mechanism to protect the consistency of task states.

[0077] After the task data transfer is complete, the cross-stage hot migration service 332 can perform task state recovery and consistency verification on the third compute node. In some cases, the cross-stage hot migration service 332 can load the received task data into the GPU memory of the third compute node. It can also reconstruct the training context, including the optimizer state and learning rate scheduler state. The cross-stage hot migration service 332 can perform consistency verification on the task data of the second and third compute nodes, for example, by comparing the hash values ​​of the two sets of task data to ensure data integrity. The cross-stage hot migration service 332 can perform one operation (e.g., one forward propagation) on both the second and third compute nodes respectively to verify whether the two sets of output results are consistent.

[0078] If the consistency check passes, the cross-stage hot migration service 332 cleans up the task data on the second compute node, releasing the computing resources allocated to the reinforcement learning task on the second compute node. The cross-stage hot migration service 332 can notify the orchestration adapter 324 to switch the scheduling target to the third compute node, terminate the reinforcement learning task process on the second compute node, and release the computing resources (e.g., GPU, video memory, and network resources) occupied on the second compute node. The cross-stage hot migration service 332 can also update the migration operation and task stage status records to complete. If there are no other tasks on the second compute node, the second compute node is marked as schedulable.

[0079] Through the above-mentioned cross-stage hot migration process, the dynamic flow and cross-stage hot migration of computing resources can be realized when the task stages of reinforcement learning tasks switch. This enables resource complementarity between task stages with high resource requirements, such as the training stage and the inference stage, which are usually not fully loaded at the same time, and significantly improves the overall utilization rate of heterogeneous computing resources.

[0080] In some scenarios, the training system can also save task data to a persistent storage system (e.g., a distributed file system) during task phase switching. The reinforcement learning training task is then resumed from the checkpoint on a third compute node. This approach is simple to implement and highly reliable. In other scenarios, the training system may not need to perform migration between compute nodes; instead, it can adjust the amount of resources allocated to the reinforcement learning task on the compute node (e.g., increasing or decreasing the number of GPUs) via a dynamic scaling controller 322.

[0081] like Figure 3 As shown, the scheduling engine 125 may include a policy simulation verification service 336. The training system can utilize the policy simulation verification service 336 to perform simulation verification on the scheduling scheme to obtain verification results. If the verification result indicates that the scheduling scheme has passed verification, the training system can schedule computing resources based on the scheduling scheme to perform reinforcement learning tasks. As an example, the policy simulation verification service 336 may be an offline verification module of the scheduling engine 125, responsible for pre-verifying and risk assessing scheduling policy changes. The policy simulation verification service 336 can perform offline simulation verification on the scheduling scheme. For example, the effectiveness of the scheduling scheme can be verified in a simulation environment using historical scheduling data. By performing sufficient offline simulation verification before the scheduling scheme is executed, the risk caused by changes in the scheduling scheme can be reduced, and the reliability of scheduling decisions can be improved. If the simulation verification result indicates that the scheduling scheme has failed verification (e.g., the prediction will lead to a significant increase in migration failure rate or intensified resource contention), the scheduling decision engine 312 can adjust the scheduling scheme or regenerate the scheduling scheme until it passes verification before actually executing the scheduling.

[0082] In some scenarios, the strategy simulation verification service 336 can also be used to perform risk assessments, evaluating the risks that changes to the scheduling scheme may bring, such as increased resource contention and higher migration failure rates. Alternatively and / or additionally, the strategy simulation verification service 336 can conduct control experiments (also known as A / B comparisons) to compare the scheduling performance of different scheduling schemes under the same load conditions. Alternatively and / or additionally, the strategy simulation verification service 336 can provide suggestions for adjusting the scheduling scheme based on the verification results.

[0083] In some scenarios, the training system can allocate computing resources to reinforcement learning tasks according to the allocation strategy indicated by the scheduling scheme. In other scenarios, the allocation strategy may include a first allocation strategy (also known as an exclusive mode), which indicates that dedicated computing resources are allocated to the reinforcement learning task. These dedicated computing resources belong exclusively to the reinforcement learning task, meaning that the task does not share these resources with other tasks during runtime. The first allocation strategy (i.e., the exclusive mode) is typically suitable for tasks with high performance stability requirements, such as large-scale distributed training and inference model training, and its resource utilization is relatively moderate.

[0084] Alternatively and / or additionally, the allocation strategy may include a second allocation strategy (also known as a sharing mode), which indicates the allocation of shared computational resources for a reinforcement learning task that shares these resources with at least one other task. The second allocation strategy (sharing mode) is typically suitable for small-scale experiments, evaluation phase tasks, or embodied intelligence tasks where environment rendering and training are performed alternately. Multiple tasks can reuse the same set of computational resources through time-slicing or spatial partitioning, resulting in relatively high resource utilization.

[0085] Alternatively and / or additionally, the allocation strategy can be a third allocation strategy (also known as a stage-level elastic mode). This third allocation strategy allows computational resources allocated for a reinforcement learning task to be migrated between multiple task stages. The third allocation strategy (stage-level elastic mode) is typically suitable for long-term reinforcement learning training and tasks with distinct stage characteristics. It supports cross-stage resource transfer (i.e., the cross-stage hot migration mechanism described above) and pursues optimal resource utilization. In some cases, the third allocation strategy can be used as the default allocation strategy. It should be understood that the above applicable scenarios are merely examples; users can also select their desired resource allocation strategy through the corresponding strategy field in the task configuration according to their actual needs.

[0086] Combination Figure 3As shown, the orchestration adapter 324 is responsible for converting the scheduling schemes output by modules such as the scheduling decision engine 312 and the dynamic scaling controller 322 into specific operations executable by the underlying resource orchestration system. In some cases, the underlying resource orchestration system may include a container orchestration platform. The orchestration adapter 324 can adopt the extension mode of the Scheduler Framework and be embedded into the scheduler component of the container orchestration platform. The scheduling logic described in this paper is implemented through the following extension points: a filtering extension point, used to implement the hardware capability screening of the heterogeneous hardware-aware scheduler 314, filtering out computing nodes that do not meet the task resource requirements; a scoring extension point, used to implement the multi-dimensional comprehensive scoring of the scheduling decision engine 312 to rank candidate computing nodes; a permission extension point, used to implement resource reservation logic to ensure the consistency of scheduling decisions; and a binding extension point, used to bind the final scheduling result to the target computing node.

[0087] In some scenarios, the orchestration adapter 324 can manage the following Custom Resource Definitions (CRDs) to represent resource objects related to reinforcement learning training within the orchestration platform. In some cases, a custom resource may include a first custom resource representing a complete reinforcement learning training task, whose core fields may include task type, task stage, resource profile, and current task stage. Alternatively and / or additionally, a custom resource may include a second custom resource representing a task stage instance within a reinforcement learning task, whose core fields may include stage type, duration, resource requirements, and status. Alternatively and / or additionally, a custom resource definition may include a third custom resource representing a pool of computational nodes available for reinforcement learning training, whose core fields may include a node selector, hardware profile, scaling strategy, and node list. Alternatively and / or additionally, a custom resource definition may include a fourth custom resource representing a cross-stage hot migration operation, whose core fields may include source node, target node, migration type (full or incremental), status, and progress. Users can select one of the first, second, or third allocation strategies described above by using the resource allocation strategy field in the custom resource definition.

[0088] In some scenarios, orchestration adapter 324 can receive scheduling schemes from scheduling decision engine 312, apply the scheduling schemes to the orchestration platform through the aforementioned extension points, and monitor the status changes of the aforementioned custom resources. When task phase sensor 304 detects a task phase switch, it coordinates dynamic scaling controller 322 to adjust resources. When cross-phase hot migration service 332 triggers cross-phase hot migration, it creates and manages the corresponding migration custom resources. Orchestration adapter 324 can also feed back the scheduling execution results to scheduling observation service 334 for link tracing. It should be understood that the specific implementation of orchestration adapter 324 described above is only an example. In other embodiments, orchestration adapter 324 can also be adapted to other types of resource orchestration systems or bare metal resource management platforms, and this document does not limit this.

[0089] Combination Figure 3 As shown, the scheduling observation service 334 is configured to perform scheduling scheme tracking and visualization. In some cases, the scheduling observation service 334 can be configured to perform scheduling scheme tracking. Specifically, the scheduling observation service 334 is configured to record the inputs (e.g., task resource profiles, task phase information, compute node status), decision-making process (e.g., scoring details for each scoring dimension), and outputs (e.g., target compute node, resource specifications) of each scheduling scheme. Alternatively and / or additionally, the scheduling observation service 334 can be configured to perform phase switch tracking, recording the time point, triggering conditions, confidence level, and resource changes before and after the phase switch. Alternatively and / or additionally, the scheduling observation service 334 can be configured to perform hot migration tracking, recording the execution status, time consumption, and results of each of the six steps of each cross-phase hot migration. Alternatively and / or additionally, the scheduling observation service 334 can be configured to monitor the utilization of computing resources, displaying the GPU utilization, memory usage, CPU utilization, network bandwidth utilization, etc., of the computing cluster and each compute node. Alternatively and / or additionally, the scheduling observation service can be configured to perform event alerts. For example, the scheduling observation service can trigger alerts when resource utilization is abnormal, scheduling fails, or migration fails.

[0090] In some cases, the data output by the scheduling observation service 334 can be used by operations and maintenance personnel to optimize scheduling strategies, or it can be used as input data for strategy simulation verification service 336 for offline analysis, thereby forming a feedback loop between scheduling decisions and actual execution effects.

[0091] A corresponding system for implementing the above methods or processes is also provided. Figure 5A block diagram of a resource scheduling apparatus 500 is shown, depending on several scenarios. Apparatus 500 can be implemented as or included in a training system, for example, it can be implemented as or included in a scheduling engine 125. The various modules / components in apparatus 500 can be implemented by hardware, software, firmware, or any combination thereof.

[0092] like Figure 5 As shown, the apparatus 500 includes: a task type determination module 510 configured to determine the task type of a reinforcement learning task for a machine learning model; a resource requirement determination module 520 configured to determine resource requirement information corresponding to the task type based on the task type, the resource requirement information describing the resource requirements of the reinforcement learning task; a scheduling scheme determination module 530 configured to determine a scheduling scheme based on the resource requirement information, the scheduling scheme indicating the computing resources used to execute the reinforcement learning task; and a scheduling module 540 configured to schedule computing resources based on the scheduling scheme to execute the reinforcement learning task.

[0093] In some cases, the task type includes at least one of the following: a first task type, wherein a reinforcement learning task of the first task type is used to train a first machine learning model, the first machine learning model being configured to generate action instructions based on observation information of a first environment, the action instructions being used to control objects in the first environment to perform actions; a second task type, wherein a reinforcement learning task of the second task type is used to train a second machine learning model, the second machine learning model being configured to generate invocation instructions for a tool, the invocation instructions being used to invoke the tool via a second environment to perform an operation corresponding to the invocation instructions; and a third task type, wherein a reinforcement learning task of the third task type is used to train a third machine learning model, the third machine learning model being configured to generate a reasoning process for a logical reasoning task.

[0094] In some cases, the reinforcement learning task includes multiple task stages, and the resource requirement information includes at least one of the following: time information indicating the duration of each of the multiple task stages; feature information indicating the computational resources required by each of the multiple task stages; preference information indicating the type of computational resources preferentially allocated to the multiple task stages; and policy information indicating a strategy for allocating computational resources to the reinforcement learning task.

[0095] In some cases, the scheduling scheme determination module 530 is further configured to: obtain the identification result of the task stage in which the reinforcement learning task is located; and determine the scheduling scheme based on the resource requirement information and the identification result.

[0096] In some cases, the scheduling scheme determination module 530 is further configured to: acquire feature information and multiple reference information, wherein the feature information indicates the running state of the reinforcement learning task and the multiple reference information respectively indicates multiple task stages of the reinforcement learning task; determine the matching degree between the feature information and the multiple reference information to obtain multiple first scores; and in response to one of the multiple first scores exceeding a first threshold, determine that the reinforcement learning task is in a first task stage or is about to switch to the first task stage, wherein the first task stage is the task stage among the multiple task stages corresponding to the first score exceeding the first threshold.

[0097] In some cases, the scheduling scheme determination module 530 is further configured to: acquire a plurality of first indicators, the plurality of first indicators indicating the usage status of the reinforcement learning task on multiple types of computing resources; and perform feature extraction based on the plurality of first indicators to acquire the feature information.

[0098] In some cases, the scheduling scheme determination module 530 is further configured to: determine that the reinforcement learning task is in the first task stage or is about to switch to the first task stage based on the fact that the first score corresponding to the first task stage exceeds the first threshold in multiple consecutive recognition cycles.

[0099] In some cases, the scheduling scheme determination module 530 is further configured to: determine a plurality of second scores corresponding to a plurality of candidate computing resources, the plurality of second scores indicating the data processing capabilities of the plurality of candidate computing resources; determine the matching degree between the plurality of candidate computing resources and the reinforcement learning task based on the plurality of second scores, the resource requirement information and the identification result, to obtain a plurality of third scores; and select computing resources from the plurality of candidate computing resources for performing the reinforcement learning task based on the plurality of third scores, to obtain the scheduling scheme.

[0100] In some cases, the second score is determined based on at least two of the following: the computing power of the candidate computing resource; the video memory capacity of the candidate computing resource; the interconnect bandwidth of the candidate computing resource; the graphics rendering capability of the candidate computing resource; and the cost-effectiveness of the candidate computing resource.

[0101] In some cases, the plurality of task phases include at least one of the following: an environment phase configured to initialize an environment instance corresponding to the reinforcement learning task; an inference phase configured to use the machine learning model to determine an operation instruction for the environment instance based on first observation information of the environment instance, and to perform the operation in the environment instance using the operation instruction to obtain second observation information of the environment instance, wherein the first observation information and the second observation information respectively indicate the environmental state of the environment instance before and after the operation; a training phase configured to perform reinforcement learning training on the machine learning model using training samples to obtain a trained machine learning model, wherein the training samples are constructed based on at least one of the operation instruction, the first observation information, and the second observation information; a reward determination phase configured to determine a reward value for the operation instruction based at least on the second observation information, wherein the reward value indicates the correctness of the operation instruction; a parameter synchronization phase configured to transfer the parameters of the trained machine learning model to at least one computing node; and an evaluation phase configured to perform an evaluation of the task quality of the reinforcement learning task to obtain an evaluation result.

[0102] In some cases, the scheduling module 540 is further configured to: in response to the recognition result indicating that the reinforcement learning task is to switch from the second task stage to the third task stage, pre-initialize the computing resources required for the third task stage on the first computing node based on the scheduling scheme; and in response to the end of the second task stage, execute the third task stage on the first computing node.

[0103] In some cases, the scheduling module 540 is further configured to: in response to a migration condition triggered by the reinforcement learning task, migrate the task data of the reinforcement learning task from a second computing node to a third computing node, wherein the migration condition includes at least the reinforcement learning task switching from a fourth task stage to a fifth task stage, the task data indicating the task state of the reinforcement learning task, the second computing node being used to execute the fourth task stage, and the third computing node being used to execute the fifth task stage; and the fifth task stage of the reinforcement learning task being executed on the third computing node according to the task state.

[0104] In some cases, the scheduling module 540 is further configured to perform at least one of the following: transferring the task data from the second computing node to the third computing node based on the migration being a full migration; and transferring incremental data in the task data from the second computing node to the third computing node based on the migration being an incremental migration, the incremental data being data that has changed in the task data since the previous migration.

[0105] In some cases, the scheduling module 540 is further configured to: perform a consistency check on the task data of the second computing node and the task data of the third computing node; and release the computing resources allocated on the second computing node for the reinforcement learning task based on the success of the consistency check.

[0106] In some cases, the scheduling module 540 is further configured to: perform simulation verification on the scheduling scheme to obtain verification results; and, based on the verification results, indicate that the scheduling scheme has passed verification, and schedule the computing resources based on the scheduling scheme to execute the reinforcement learning task.

[0107] In some cases, the scheduling module 540 is further configured to allocate computing resources to the reinforcement learning task based on the scheduling scheme according to at least one of the following strategies: a first allocation strategy, which indicates that dedicated computing resources are allocated to the reinforcement learning task, the dedicated computing resources being exclusively for the reinforcement learning task; a second allocation strategy, which indicates that shared computing resources are allocated to the reinforcement learning task, the reinforcement learning task sharing the shared computing resources with at least one other task; and a third allocation strategy, which indicates that the computing resources allocated to the reinforcement learning task are allowed to migrate between multiple task stages.

[0108] The modules included in device 500 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some cases, one or more modules can be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the units in device 500 can be implemented at least partially by one or more hardware logic components. By way of example and not limitation, exemplary types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard parts (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), and so on.

[0109] Figure 6 A block diagram of an electronic device 600 in which one or more examples may be implemented is shown. It should be understood that... Figure 6 The electronic device 600 shown is merely exemplary and should not be construed as limiting the functionality and scope of the examples described herein. Figure 6 The electronic device 600 shown can be used to implement the training system discussed above.

[0110] like Figure 6 As shown, electronic device 600 is in the form of a general-purpose electronic device. Components of electronic device 600 may include, but are not limited to, one or more processing units or processors 610, memory 620, storage device 630, one or more communication units 640, one or more input devices 650, and one or more output devices 660. Processor 610 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 620. In a multiprocessor system, multiple processors execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 600.

[0111] Electronic device 600 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 600, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 620 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof). Storage device 630 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data and can be accessed within electronic device 600.

[0112] Electronic device 600 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 6 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 620 may include computer program product 625 having one or more program modules configured to perform various methods or actions of various examples.

[0113] The communication unit 640 enables communication with other electronic devices via a communication medium. Additionally, the functionality of the components of the electronic device 600 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 600 can operate in a networked environment using logical connections to one or more other servers, networked personal computers, or another network node.

[0114] Input device 650 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 660 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 600 can also communicate with one or more external devices (not shown) via communication unit 640 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 600, or with any device that enables electronic device 600 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via an input / output (I / O) interface (not shown).

[0115] A computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. A computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0116] The flowcharts and / or block diagrams of the methods, apparatus, devices, and computer program products referred to herein describe various aspects. It should be understood that each block of the flowcharts and / or block diagrams, as well as combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0117] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0118] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0119] The flowcharts and block diagrams in the accompanying figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products under various scenarios. In this respect, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the figures. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0120] Various examples have been described above. The foregoing descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for resource scheduling, comprising: Determine the task type of the reinforcement learning task for the machine learning model; Based on the task type, resource requirement information corresponding to the task type is determined, and the resource requirement information is used to describe the computing resources required for the reinforcement learning task; Based on the resource requirement information, a scheduling scheme is determined, wherein the scheduling scheme indicates the computing resources used to perform the reinforcement learning task; and Based on the scheduling scheme, computing resources are scheduled to execute the reinforcement learning task.

2. The method of claim 1, wherein the task type includes at least one of the following: A first task type, wherein a reinforcement learning task of the first task type is used to train a first machine learning model, the first machine learning model being configured to generate action instructions based on observation information of a first environment, the action instructions being used to control objects in the first environment to perform actions; The second task type, wherein the reinforcement learning task of the second task type is used to train a second machine learning model, the second machine learning model being configured to generate invocation instructions for a tool, the invocation instructions being used to invoke the tool via a second environment to perform an operation corresponding to the invocation instructions; as well as The third task type, wherein the reinforcement learning task of the third task type is used to train a third machine learning model, the third machine learning model being configured to generate reasoning processes for logical reasoning tasks.

3. The method of claim 1, wherein the reinforcement learning task comprises multiple task stages, and wherein the resource requirement information comprises at least one of the following: Time information, which indicates the duration of each of the plurality of task phases; Feature information, which indicates the computing resources required by each of the plurality of task stages; Preference information, which indicates the type of computing resources that are preferentially allocated for the plurality of task phases; as well as Policy information, which indicates a strategy for allocating computational resources to the reinforcement learning task.

4. The method of claim 1, wherein determining the scheduling scheme comprises: Obtain the identification result of the task stage in which the reinforcement learning task is located; as well as Based on the resource demand information and the identification results, the scheduling scheme is determined.

5. The method according to claim 4, wherein obtaining the identification result includes: Acquire feature information and multiple reference information, wherein the feature information indicates the running state of the reinforcement learning task, and the multiple reference information respectively indicates multiple task stages of the reinforcement learning task; Determine the matching degree between the feature information and the multiple reference information to obtain multiple first scores; as well as In response to one of the plurality of first scores exceeding a first threshold, it is determined that the reinforcement learning task is in a first task stage or is about to switch to the first task stage, wherein the first task stage is the task stage among the plurality of task stages corresponding to the first score that exceeds the first threshold.

6. The method according to claim 5, wherein obtaining the feature information includes: Acquire multiple first indicators, which indicate the usage status of multiple types of computing resources by the reinforcement learning task; as well as Feature extraction is performed on the plurality of first indicators to obtain the feature information.

7. The method of claim 5, wherein determining that the reinforcement learning task is in the first task stage comprises: Based on the fact that the first score corresponding to the first task stage exceeds the first threshold in multiple consecutive recognition cycles, it is determined that the reinforcement learning task is in the first task stage or is about to switch to the first task stage.

8. The method of claim 4, wherein determining the scheduling scheme comprises: Determine multiple second scores corresponding to multiple candidate computing resources, wherein the multiple second scores indicate the data processing capabilities of the multiple candidate computing resources; Based on the multiple second scores, the resource requirement information, and the identification results, the matching degree between the multiple candidate computing resources and the reinforcement learning task is determined to obtain multiple third scores; Based on the multiple third scores, computing resources for performing the reinforcement learning task are selected from the multiple candidate computing resources to obtain the scheduling scheme.

9. The method of claim 8, wherein the second score is determined based on at least two of the following: The computing power of the candidate computing resources; The video memory capacity of the candidate computing resources; The interconnect bandwidth of the candidate computing resources; The graphics rendering capabilities of the candidate computing resources; as well as The cost-effectiveness of the candidate computing resources.

10. The method of claim 4, wherein the plurality of task phases comprises at least one of the following: The environment phase is configured to initialize an environment instance corresponding to the reinforcement learning task. In the inference phase, the inference phase is configured to use the machine learning model to determine an operation instruction for the environment instance based on a first observation information of the environment instance, and to use the operation instruction to perform an operation in the environment instance to obtain a second observation information of the environment instance, wherein the first observation information and the second observation information respectively indicate the environmental state of the environment instance before and after the operation; In the training phase, the training phase is configured to perform reinforcement learning training on the machine learning model using training samples to obtain a trained machine learning model, the training samples being constructed based on at least one of the operation instructions, the first observation information, and the second observation information; In the reward determination phase, the reward determination phase is configured to determine a reward value for the operation instruction based at least on the second observation information, the reward value indicating the correctness of the operation instruction; The parameter synchronization phase is configured to transfer the parameters of the trained machine learning model to at least one computing node. as well as The evaluation phase is configured to perform an evaluation of the task quality of the reinforcement learning task to obtain evaluation results.

11. The method of claim 4, wherein scheduling computing resources based on the scheduling scheme comprises: In response to the recognition result indicating that the reinforcement learning task is about to switch from the second task stage to the third task stage, the computing resources required for the third task stage are pre-initialized on the first computing node based on the scheduling scheme; as well as In response to the end of the second task phase, the third task phase is executed on the first computing node.

12. The method of claim 4, wherein scheduling computing resources based on the scheduling scheme comprises: In response to the reinforcement learning task triggering a migration condition, the task data of the reinforcement learning task is migrated from the second computing node to the third computing node, wherein the migration condition includes at least the reinforcement learning task switching from the fourth task stage to the fifth task stage, the task data indicating the task state of the reinforcement learning task, the second computing node being used to execute the fourth task stage, and the third computing node being used to execute the fifth task stage; and The fifth task phase of the reinforcement learning task is executed on the third computing node according to the task state.

13. The method of claim 12, wherein migrating the task data of the reinforcement learning task from the second computing node to the third computing node comprises at least one of the following: Based on the fact that the migration is a full migration, the task data is transferred from the second computing node to the third computing node; and Since the migration is an incremental migration, the incremental data in the task data is transferred from the second computing node to the third computing node. The incremental data is the data that has changed in the task data after the previous migration.

14. The method of claim 13, wherein migrating the task data of the reinforcement learning task from the second computing node to the third computing node further comprises: Perform a consistency check on the task data of the second computing node and the task data of the third computing node; as well as Based on the successful consistency check, the computing resources allocated to the reinforcement learning task on the second computing node are released.

15. The method of claim 1, wherein scheduling computing resources based on the scheduling scheme comprises: The scheduling scheme is simulated and verified to obtain the verification results; as well as Based on the verification result, the scheduling scheme is verified as successful, and the computing resources are scheduled according to the scheduling scheme to execute the reinforcement learning task.

16. The method of claim 1, wherein scheduling a computational resource based on the scheduling scheme comprises: Based on the scheduling scheme, computational resources are allocated to the reinforcement learning task according to at least one of the following strategies: A first allocation strategy, wherein the first allocation strategy indicates that dedicated computing resources are allocated to the reinforcement learning task, and the dedicated computing resources are exclusively for the reinforcement learning task; A second allocation strategy is indicated for allocating shared computing resources to the reinforcement learning task, which shares the shared computing resources with at least one other task. as well as A third allocation strategy indicates that computational resources allocated for the reinforcement learning task can be migrated between multiple task phases.

17. An apparatus for resource scheduling, comprising: The task type determination module is configured to determine the task type of the reinforcement learning task of the machine learning model; The resource requirement determination module is configured to determine resource requirement information corresponding to the task type based on the task type, wherein the resource requirement information is used to describe the resource requirements of the reinforcement learning task. The scheduling scheme determination module is configured to determine a scheduling scheme based on the resource requirement information, wherein the scheduling scheme indicates the computing resources used to perform the reinforcement learning task; as well as The scheduling module is configured to schedule computing resources based on the scheduling scheme to execute the reinforcement learning task.

18. An electronic device, comprising: At least one processing unit; as well as At least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 16 when executed by the at least one processing unit.

19. A computer-readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement the method according to any one of claims 1 to 16.

20. A computer program product tangibly stored in a computer storage medium and comprising computer-executable instructions that, when executed by a device, cause the device to perform the method according to any one of claims 1 to 16.

Images