Model reinforcement training system, resource scheduling method and apparatus

Abstract

The present disclosure relates to a model reinforcement training system, a resource scheduling method and device. The model reinforcement training process includes one or more training rounds, and each training round includes multiple computing stages. The resource scheduling method includes: monitoring the running state of at least one first computing stage in the multiple computing stages to obtain first running state information, the first running state information including a first remaining request quantity; determining a scalable resource of the first computing stage based on the first remaining request quantity, a target batch size of each first computing instance in the first computing stage, and a resource amount occupied by each first computing instance; and migrating at least part of the scalable resource to at least one second computing stage in the multiple computing stages. In this way, while maintaining a high throughput index of the first computing stage, the first computing stage can be scaled down, and resources can be allocated to the second computing stage, thereby improving the overall resource utilization of the system.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence, and in particular to a model reinforcement training system, resource scheduling method and apparatus. Background Technology

[0002] Large Language Models (LLMs) are deep learning-based artificial intelligence models, typically employing the Transformer architecture. They learn the statistical patterns and semantic information of language through self-supervised pre-training on massive amounts of text data, thereby acquiring powerful language generation, understanding, and reasoning capabilities.

[0003] The training process of LLM is usually divided into two main stages: the first is the pre-training stage, in which the model learns language knowledge from scratch; the second is the fine-tuning stage, in which reinforcement learning from human feedback (RLHF) is a key step, aiming to make the model's output more in line with human values ​​and preferences.

[0004] Most large language model reinforcement training systems employ fully sequential, co-located synchronous scheduling. Taking the reinforcement learning algorithm GRPO (Group Relative Policy Optimization) as an example, a training epoch is divided into three phases: generation, inference, and training. Most related systems allow all GPUs to first enter the generation phase; after all GPUs have completed this phase, the latter two phases are run in the same manner. Summary of the Invention

[0005] While the synchronous scheduling method in related technologies has clear logic and can guarantee training stability, its resource utilization is low.

[0006] Based on this, this disclosure presents a resource scheduling scheme for a model reinforcement training system to improve resource utilization during the model reinforcement training process.

[0007] According to a first aspect of this disclosure, a resource scheduling method for a model reinforcement training process is provided, wherein the model reinforcement training process includes one or more training rounds, each training round includes multiple computation stages, and the method includes: monitoring the running status of at least one first computation stage among the multiple computation stages to obtain first running status information, the first running status information including a first remaining number of requests; determining scalable resources for the first computation stage based on the first remaining number of requests, the target batch size of each first computation instance in the first computation stage, and the amount of resources occupied by each first computation instance; and migrating at least a portion of the scalable resources to at least one second computation stage among the multiple computation stages.

[0008] Optionally, based on the first remaining request count, the target batch size of each first computing instance in the first computing phase, and the amount of resources occupied by each first computing instance, the scalable resources for the first computing phase are determined, including: determining the minimum set of first computing instances to be retained based on the first remaining request count, the target batch size of each first computing instance, and the amount of resources occupied by each first computing instance, wherein the sum of the target batch sizes of all first computing instances in the first computing instance set is greater than or equal to the first remaining request count; and determining scalable resources based on the first computing instance set, wherein the scalable resources include the resources remaining after deducting the resources occupied by all first computing instances in the first computing phase from the resources occupied by all first computing instances in the first computing phase.

[0009] Optionally, the first running status information also includes the distribution of the number of remaining requests across different first computing instances. Determining the minimum set of first computing instances to be retained includes: selecting first computing instances one by one from all the current first computing instances in the first computing phase in descending order of the number of remaining requests, until the sum of the target batch sizes of the cumulatively selected first computing instances is greater than or equal to the number of remaining requests for the first time, at which point selection stops. The cumulative set of selected first computing instances is the minimum set of first computing instances to be retained.

[0010] Optionally, migrating at least some scalable resources to at least one of the plurality of computing stages includes: unloading at least some of the first computing instances outside the first computing instance set; and allocating the resources occupied by the unloaded at least some of the first computing instances in the scalable resources to the second computing stage.

[0011] Optionally, migrating at least some scalable resources to at least one of the plurality of computing stages to a second computing stage includes: obtaining a second remaining number of requests in the second computing stage; determining resource constraints for scaling up the second computing stage based on the second remaining number of requests and the amount of resources required by a single second computing instance in the second computing stage; and allocating the scalable resources that meet the resource constraints to the second computing stage.

[0012] Optionally, the method further includes: using a first time consumption model to predict the time consumption of the first computing instance under different batch sizes; calculating the throughput index of the first computing instance under different batch sizes based on the predicted time consumption; and determining the optimal batch size for the throughput index as the target batch size.

[0013] Optionally, the method further includes: pre-obtaining performance statistics of computing instances in at least one computing stage under different resource configurations, wherein the resource configuration represents the amount of resources occupied by a single computing instance, and the performance statistics include at least time consumption data; constructing a time consumption model under different resource configurations based on the performance statistics; and determining the optimal resource configuration of computing instances in the at least one computing stage based on the initial resource quantity and request scale of the computing stage, using the time consumption model, before the model reinforcement training process begins, to initialize the deployment of the at least one computing stage.

[0014] Optionally, the plurality of computational stages include a generation stage, an inference stage, and a training stage, wherein the first computational stage includes the generation stage, the second computational stage includes the training stage, and the method further includes: at the beginning of each training round, allocating a fixed first portion of resources from the resource pool to the inference stage, and allocating a second portion of resources other than the first portion of resources from the resource pool to the generation stage, wherein the amount of the second portion of resources is greater than that of the first portion of resources.

[0015] Optionally, the training samples for each training round are divided into multiple global batches during the training phase. Before the first global batch begins, in response to the first remaining request quantity being lower than a first threshold, the steps of determining the scalable resources of the first computing phase and migrating at least some of the scalable resources to at least one second computing phase among the multiple computing phases are executed. Before the start of each subsequent global batch, the steps of determining the scalable resources of the first computing phase and migrating at least some of the scalable resources to at least one second computing phase among the multiple computing phases are executed.

[0016] According to a second aspect of this disclosure, a resource scheduling apparatus for a model reinforcement training process is provided, wherein the model reinforcement training process includes one or more training rounds, each training round including multiple computation stages, comprising: a monitor configured to monitor the running status of at least one first computation stage among the multiple computation stages, and obtain first running status information, the first running status information including a first remaining number of requests; a scheduler configured to determine scalable resources of the first computation stage based on the first remaining number of requests, the target batch size of each first computation instance in the first computation stage, and the amount of resources occupied by each first computation instance; and a controller configured to migrate at least a portion of the scalable resources to at least one second computation stage among the multiple computation stages.

[0017] According to a third aspect of this disclosure, a model reinforcement training system is provided, wherein the model reinforcement training process includes one or more training rounds, each training round includes multiple computational stages, the system includes a scheduling device and an execution device, the execution device includes multiple computational modules, each computational module corresponds to one computational stage, the scheduling device monitors the operating status of at least one first computational module among the multiple computational modules to obtain first operating status information of the first computational module, the first operating status information includes a first remaining number of requests, the scheduling device determines the scalable resources of the first computational stage based on the first remaining number of requests, the target batch size of each first computational instance in the first computational stage, and the amount of resources occupied by each first computational instance, the scheduling device controls the execution device to migrate at least a portion of the scalable resources to at least one second computational module among the multiple computational modules.

[0018] According to a third aspect of this disclosure, a computer program product is provided, including executable code that, when executed by a processor of an electronic device, causes the processor to perform the method described in the first aspect above.

[0019] According to a fourth aspect of this disclosure, a non-transitory machine-readable storage medium is provided, on which executable code is stored, which, when executed by a processor of an electronic device, causes the processor to perform the method described in the first aspect above.

[0020] During model reinforcement training, this disclosure determines the scalable resources for the first computation stage based on the number of remaining requests in the first computation stage (e.g., the generation stage), the target batch size of each first computation instance in the first computation stage, and the amount of resources occupied by each first computation instance. At least some of these scalable resources are then migrated to the second computation stage (e.g., the training stage). Thus, while maintaining a high throughput in the first computation stage, the first computation stage can be scaled down, freeing up resources for the second computation stage, thereby improving the overall resource utilization of the system. Attached Figure Description

[0021] The above and other objects, features and advantages of this disclosure will become more apparent from the more detailed description of exemplary embodiments thereof taken in conjunction with the accompanying drawings, wherein like reference numerals generally denote like parts.

[0022] Figures 1A to 1C This demonstrates a fully serialized, co-located synchronous scheduling method.

[0023] Figure 2 A schematic diagram of a resource scheduling scheme according to some embodiments of the present disclosure is shown.

[0024] Figure 3 A schematic flowchart of a resource scheduling method for a model reinforcement training process according to some embodiments of the present disclosure is shown.

[0025] Figure 4 A schematic diagram of the architecture of a scheduling system according to some embodiments of the present disclosure is shown. Detailed Implementation

[0026] Preferred embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0027] In this specification, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, B exists alone, or A and B exist simultaneously. Furthermore, unless otherwise stated, the term "multiple" means two or more.

[0028] Furthermore, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the indicated technical feature. Therefore, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. The terms "comprising," "including," "having," and their variations all mean "including but not limited to," unless otherwise specifically emphasized.

[0029] As the capabilities of basic models become increasingly powerful, reinforcement learning has gradually evolved from simply aligning with human preferences to specifically enhancing the capabilities of models (such as large language models). Domain-specific reinforcement learning can improve a model's performance on related problems. For example, by applying reinforcement learning to datasets of mathematical problems, the model's ability to solve mathematical problems can be improved, thus creating a mathematical problem-solving model.

[0030] The model trained using reinforcement learning algorithms disclosed herein can be either a model for human preference alignment (i.e., output that is more in line with human values ​​and preferences) or a model with strong problem-solving capabilities in a specific domain (such as a mathematical problem-solving model).

[0031] In the process of training a model based on a reinforcement learning algorithm, a training epoch generally includes a generation phase, an inference phase, and a training phase. The following is a brief explanation of these three phases using the GRPO (Group Relative Policy Optimization) algorithm as an example. It should be understood that this disclosure does not involve any improvement to the reinforcement learning algorithm itself; therefore, details regarding these three phases can be found in related technologies. Furthermore, this disclosure does not limit the reinforcement learning algorithm used; it can be the GRPO algorithm, or the PPO (Proximal Policy Optimization) algorithm, the DPO (Direct Preference Optimization) algorithm, etc.

[0032] 1. Generation stage

[0033] During the generation phase, the actor model generates responses for the current batch of input. Ultimately, each prompt word and its corresponding response constitute the training samples for this iteration, referred to as a rollout in the context of reinforcement learning. To improve sample efficiency, a large number of samples are typically generated in a single iteration. The actor model is also the model to be trained.

[0034] In reinforcement learning model training, this step is typically accomplished using a dedicated inference engine (such as SGLang, vLLM, and other open-source projects). LLM inference is an autoregressive process. The entire inference process can be divided into two stages: prefilling and decoding. The prefilling stage processes the entire input word (token) sequence. In the decoding stage, each decoding operation produces an output word. This word is then used as input for the next decoding iteration, generating a new word. This process iterates continuously until the model outputs a terminator, resulting in an output word sequence for each input word sequence.

[0035] 2. Reasoning Stage

[0036] The inference phase of reinforcement learning differs from model inference. It involves the reference model and the model to be trained performing forward propagation computation on the generated samples from the previous phase. That is, the generated samples from the previous phase are input into the reference model and the model to be trained, respectively. Here, "generated samples" refers to the output word sequence generated in the "generation phase." The results of the inference phase can be used to calculate the KL divergence loss term in the loss function during subsequent training phases. In some alternative embodiments, the tasks of the inference phase can also be performed in the generation phase.

[0037] 3. Training Phase

[0038] The training phase typically utilizes relevant training engines such as Megatron-LM and PyTorch FSDP. Unlike traditional LLM training, all samples in a training epoch are divided into multiple mini-batches, and the model weights need to be updated after each mini-batch is processed. Specifically: for each mini-batch, the actor model first performs forward propagation, then calculates the loss function based on the results of the inference phase, and updates the parameters through backpropagation. After the actor model has completed training for all mini-batches, it will use the latest parameters for the next generation phase. The mini-batch is also referred to as the global batch below.

[0039] Figures 1A to 1C This demonstrates a fully serialized, co-located synchronous scheduling method.

[0040] See Figure 1A Within each training epoch, all computing resources (GPU resources) in the resource pool are first allocated to the generation phase. That is, the generation phase is executed first using all available computing resources. See also... Figure 1B After all requests in the generation phase have been executed, all computing resources in the resource pool are then allocated to the inference phase. That is, the inference phase is then executed using all available computing resources. See also... Figure 1CAfter all requests in the inference phase have been executed, all computing resources in the resource pool are allocated to the training phase. In other words, the training phase is executed using all available computing resources.

[0041] However, resource utilization varies across different stages of reinforcement learning. Taking the generation stage as an example, since LLM inference is an autoregressive process, it often takes up the most time. To fully utilize computational resources, the generation stage groups several sequences into a batch, generating the next word for each sequence in the batch with each decoding iteration. However, the output sequence length for each input sequence is uncertain and varies. As the inference engine instance infers the input sequences in its assigned batch, some input sequences may terminate prematurely, causing the number of input sequences in the batch to gradually decrease. Since the number of input sequences in a training epoch is limited, it's impossible to replenish the input sequences in the batch. This leads to low resource utilization in the later stages of the inference process, and sequences that have completed the generation stage cannot proceed to the next stage due to the synchronous execution mode; this phenomenon is generally referred to as the long-tail effect.

[0042] and Figures 1A to 1C Unlike the fully serialized, co-located synchronous scheduling method shown, this disclosure proposes a dynamic resource scheduling scheme. Figure 2 A schematic diagram of a resource scheduling scheme according to some embodiments of the present disclosure is shown.

[0043] See Figure 2 In this disclosure, multiple computational phases (such as generation, inference, and training phases) within a single training epoch are performed in parallel. That is, the generation, inference, and training phases can be performed simultaneously.

[0044] Taking a single training round with 10,000 samples as an example, the number of requests (i.e., the number of input sequences) to be processed in the generation phase is 10,000. According to... Figures 1A to 1C The fully serialized, co-located synchronous scheduling method shown requires waiting for the generation phase to process all 10,000 requests before entering the inference phase. In the embodiments of this specification, the inference phase can begin without waiting for the generation phase to process all 10,000 requests; for example, the generated samples (output word sequences) generated during the generation phase can be passed to the inference phase in real time. Similarly, the training phase can begin without both the generation and inference phases being completed; for example, it can begin once a global batch of samples is ready. Therefore, the generation, inference, and training phases can be performed simultaneously during the execution of a single training round.

[0045] Based on this training method that involves multiple computational stages running in parallel, this disclosure introduces a dynamic monitoring and scheduling mechanism to implement a dynamic scaling-up and scaling-down mechanism for multiple computational stages in a single training round. This allows for scaling down computational stages with low resource utilization (such as the generation stage) at appropriate times, freeing up resources for other computational stages (such as the training stage), thereby improving the overall resource utilization and throughput of the system.

[0046] Figure 3 A schematic flowchart of a resource scheduling method for a model reinforcement training process according to some embodiments of the present disclosure is shown. Exemplary, Figure 3 The method shown can be executed by a specific device in the model reinforcement training system (e.g., a software module deployed on the CPU side, such as the scheduling plane described below).

[0047] See Figure 3 In step S310, the running status of at least one first computing stage among multiple computing stages is monitored to obtain the first running status information of the first computing stage.

[0048] The model reinforcement training process may include one or more training rounds. Each training round may include multiple computational phases, also known as processing phases. These computational phases may include, for example, the generation phase, inference phase, and training phase mentioned above.

[0049] The first computation phase can be considered a scalable computation phase among multiple computation phases. Theoretically, each computation phase can be considered a potentially scalable computation phase. In actual training, the input of subsequent computation phases (such as the inference phase and the training phase) depends on the output of the first computation phase (such as the generation phase). Therefore, more computational resources can be allocated to the first computation phase initially, while fewer resources can be allocated to other computation phases. For example, most or even all of the computational resources can be allocated to the first computation phase initially. Here, computational resources can refer to GPU resources.

[0050] In other words, the computation phase that is logically the first to be executed during training is initially allocated a larger amount of resources. Therefore, in some embodiments, at least the computation phase that is logically the first to be executed during training (such as the generation phase) can be considered as the first computation phase to be scaled down. That is, at least one first computation phase includes the first computation phase to be executed.

[0051] The first operational status information includes the first remaining request count. The first remaining request count refers to the number of requests that have not yet been processed in the first computation stage. Taking the first computation stage as the generation stage as an example, the first remaining request count is also the number of input sequences that have not been processed.

[0052] The number of remaining requests can characterize the current load of the first computing phase. In some embodiments, based on the monitored number of remaining requests, it can be first determined whether the current load of the first computing phase has decreased significantly, such as determining whether the decrease rate of the current load of the first computing phase exceeds a threshold. For example, the decrease rate of the current load can be determined by judging the decrease rate of the currently monitored number of remaining requests. The decrease rate of the current load can be equal to the decrease rate of the number of remaining requests. The decrease rate of the number of remaining requests can refer to the decrease rate of the currently monitored number of remaining requests relative to the number of requests monitored at the beginning of the first computing phase or under the condition of migrating scalable resources. If the current load decreases significantly, for example, by more than 50%, it can be considered that the resource utilization of the first computing phase is low at this time, and scaling down can be performed to release idle resources and support the scaling up of other phases.

[0053] If it is determined that a reduction in capacity is required, steps S320 and S330 can be executed.

[0054] In step S320, the scalable resources of the first computing stage are determined based on the first remaining request quantity, the target batch size of each first computing instance in the first computing stage, and the amount of resources occupied by each first computing instance.

[0055] Taking the first computation phase as the generation phase as an example, the first computation instance is also the inference engine instance. The target batch size refers to the batch size when the throughput metric (throughput or throughput rate) of the first computation instance reaches a high level (such as the maximum value). The batch size is also the number of requests processed simultaneously by a single computation instance. The resource consumption of the first computation instance can refer to the number of computing devices (such as GPUs) used to deploy the first computation instance.

[0056] It should be noted that the computing resources used in the model reinforcement training process can be a cluster of computing devices consisting of identical computing devices (such as a homogeneous GPU cluster), or a cluster of computing devices containing different models, generations, or architectures (such as a heterogeneous GPU cluster). Accordingly, the amount of resources used by different computing instances in the same computing phase (or, in other words, the resource configuration of different computing instances in the same computing phase) can be the same or different. Similarly, the target batch size of different computing instances in the same computing phase can be the same or different.

[0057] In some exemplary implementations, the amount of resources (i.e., resource configuration) used by different first computing instances in the first computing phase can be the same; for example, each first computing instance can use four GPUs of the same type. The target batch size of first computing instances with the same resource configuration is also the same.

[0058] When performing step S320, the scalable resources can be determined with the goal of "releasing more computing resources (GPU resources) while maintaining the highest possible throughput."

[0059] It should be understood that the target batch size of the first computing instance, the amount of resources occupied by the first computing instance, and other data related to resource scheduling (such as the number of first computing instances) can all be obtained through monitoring or other means.

[0060] In some exemplary implementations, the minimum set of first computing instances that need to be retained can first be determined based on the first remaining request quantity, the target batch size of each first computing instance, and the amount of resources occupied by each first computing instance.

[0061] The sum of the target batch sizes of all first compute instances in the first compute instance set should be greater than or equal to the number of first remaining requests, in order to scale down while maintaining high throughput. The minimum set of first compute instances to be retained can be, for example, the set of first compute instances with the smallest total resource size that meets the condition that "the sum of the target batch sizes of all first compute instances in the set is greater than or equal to the number of first remaining requests".

[0062] Next, scalable resources can be determined based on the first set of computing instances. Scalable resources include the resources remaining after deducting the resources occupied by all first computing instances in the first computing phase from the resources occupied by all first computing instances in the first computing instance set. Therefore, the amount of scalable resources is equal to the sum of the resources occupied by all first computing instances in the first computing phase minus the sum of the resources occupied by all first computing instances in the first computing instance set. Wherein, given that the resource amount occupied by each first computing instance is known, both the sum of the resources occupied by all first computing instances in the first computing phase and the sum of the resources occupied by all first computing instances in the first computing instance set can be calculated.

[0063] Taking the example of different first compute instances having the same resource configuration (i.e., the amount of resources occupied by different first compute instances), the minimum number of first compute instances that need to be retained can be determined based on the number of remaining requests and the target batch size of each first compute instance. For example, the minimum number of first compute instances that need to be retained can be equal to the rounded-up result of the number of remaining requests divided by the target batch size. Selecting the minimum number of first compute instances that need to be retained from all first compute instances constitutes the set of first compute instances that need to be retained at the minimum. Next, based on the first compute instances and the minimum number of first compute instances that need to be retained, the unloadable first compute instances can be determined. Unloadable first compute instances include the compute instances remaining after removing the minimum number of first compute instances that need to be retained from all first compute instances. The number of unloadable first compute instances is equal to the total number of first compute instances minus the minimum number of first compute instances that need to be retained. Based on the number of unloadable first compute instances and the amount of resources occupied by a single first compute instance, the amount of resources that can be scaled down in the first compute stage can be determined. For example, the amount of resources that can be scaled down is equal to the number of unloadable first compute instances multiplied by the amount of resources occupied by a single first compute instance.

[0064] For example, suppose the optimal batch size (i.e., the target batch size) for each first compute instance to reach maximum throughput is 100, and the total number of first compute instances is 100. Initially, there are 10,000 requests being processed. After a period of time, 3,000 requests remain unprocessed, resulting in low resource utilization. Based on the optimal batch size of 100 for each first compute instance to reach maximum throughput, only 30 first compute instances are needed to maximize throughput while freeing up the most GPU resources. Therefore, the resources freed up by unloading 70 first compute instances are the scalable resources.

[0065] In some embodiments, the first running status information also includes the distribution of the number of remaining requests across different first computing instances; that is, the current number of remaining requests in each first computing instance can be recorded in the first running status information. In this case, the minimum set of first computing instances to be retained can be determined according to the rule of prioritizing the retention of first computing instances with a larger number of remaining requests. In other words, when determining the set of first computing instances to be retained, on the one hand, priority should be given to retaining first computing instances with a larger number of remaining requests (e.g., those ranked higher in terms of the number of remaining requests); on the other hand, it should be ensured that the sum of the target batch sizes of the retained first computing instances is greater than or equal to the number of remaining requests. By simultaneously considering both aspects, the set of first computing instances with the smallest total resource volume can be determined.

[0066] For example, all the first computing instances in the first computing phase can be ranked in descending order of the number of remaining requests, and then selected one by one in order of ranking until the sum of the target batch size of the selected first computing instances is greater than or equal to the number of remaining requests for the first time. At this point, the selection stops, and the set of selected first computing instances is the minimum set of first computing instances that need to be retained.

[0067] In step S330, at least some of the scalable resources are migrated to at least one second computing stage among the multiple computing stages.

[0068] The second computation stage is a scalable computation stage among multiple computation stages. Theoretically, each computation stage can be considered a potentially scalable computation stage. Considering that in practical applications, more computational resources are usually allocated to the first computation stage, while fewer are allocated to other computation stages, the computation stages following the first computation stage can be considered scalable second computation stages. Taking multiple computation stages including the generation stage, inference stage, and training stage as an example, the inference stage and the training stage can both be considered scalable second computation stages.

[0069] Migrating at least a portion of the scalable resources from the first computing phase to the second computing phase involves two actions: resource scaling down in the first computing phase and resource scaling up in the second computing phase. Resource scaling down in the first computing phase involves unloading a portion of the first computing instances from the first computing phase to free up resources. Resource scaling up in the second computing phase involves deploying second computing instances based on the freed-up resources to perform computing in the second phase.

[0070] In some exemplary embodiments, after determining the minimum set of first computing instances that need to be retained, at least a portion of the first computing instances outside the first set of computing instances can be unloaded to free up as much computing resources as possible. Then, the resources occupied by the aforementioned at least a portion of the unloaded first computing instances in the scalable resources can be allocated to the second computing stage to expand the capacity of the second computing stage.

[0071] An exemplary implementation process for expanding the capacity of the second computing stage is as follows.

[0072] First, the number of remaining requests in the second phase can be obtained. For example, the running status of the second computation phase can be monitored to obtain the second running status information of the second computation phase. The second running status information includes the number of remaining requests.

[0073] Taking the second computation phase as the training phase as an example, the second computation instance can also be called the training engine instance. When the training samples in each training round are evenly divided into multiple global batches during the training phase, the number of second remaining requests is equal to the size of the global batches. At this point, the number of second remaining requests is fixed and does not need to be monitored.

[0074] Next, based on the number of remaining requests and the amount of resources required by a single second computing instance in the second computing phase, the resource constraints during the expansion of the second computing phase can be determined.

[0075] Resource constraints characterize the resource limitations that should be followed when scaling up the second computing phase. Taking the example that no resources were allocated to the second computing phase before scaling up, resource constraints can include limitations on the amount of resources required for scaling up. These constraints can be configured according to actual business needs. For example, the acceptable amount of scaling up resources for the second computing phase, i.e., the number of computing devices (GPUs), should be divisible by the number of remaining second computing requests, and should be an integer multiple of the resource amount occupied by a single second computing instance.

[0076] For example, assuming the second remaining number of requests is 10,000, the number of training engine instances should be divisible by 10,000. This means that only resource allocations corresponding to a number of training engine instances divisible by 100 or 200 (i.e., 100 or 200) can be selected. Furthermore, a training engine instance may occupy several GPUs; for example, a training engine instance must have 4 GPUs. Therefore, the scalable resources should also be a multiple of 4 GPUs. Combining these two aspects, the resource constraints in this example are: the acceptable scalable resources should be a multiple of 4 GPUs, and the total number of GPUs should be divisible by 40,000 (because 4 GPUs constitute one training engine instance).

[0077] After determining the resource constraints for scaling up the second computing phase, resources that meet the resource constraints from the scalable resources can be allocated to the second computing phase. For example, the maximum amount of resources that meet the resource constraints from the scalable resources (such as the resources released after unloading the first computing instance as mentioned above) can be allocated to the second computing phase. In this way, the second computing phase can be scaled up to the maximum extent.

[0078] In some embodiments, during execution Figure 3 Prior to the method shown, a pre-built first time consumption model can be used to predict the time consumption of the first computing instance under different batch sizes. Based on the predicted time consumption, the throughput metrics (such as throughput or throughput rate) of the first computing instance under different batch sizes can be calculated, and the batch size with the optimal throughput metrics can be determined as the target batch size.

[0079] The first time-consuming model can be viewed as a time-consuming model built for the first computing instance. The first time-consuming model can predict the time required for the first computing instance to execute requests of that batch size based on the input batch size (i.e., the number of requests).

[0080] For example, a formula model can be performed based on the request size, model size, and hardware metrics of the first computing instance under the corresponding resource configuration, and the actual runtime can be used for fitting to obtain the time consumption model. Here, the request size and model size can be used to measure the computational load, and the hardware metrics of the first computing instance under the corresponding resource configuration can be used to measure the computing power. Therefore, based on these parameters, a first time consumption model can be obtained through formula modeling.

[0081] In some embodiments, during execution Figure 3 Prior to the method described, performance statistics for computing instances under different resource configurations can be obtained in advance for at least one computing stage. Resource configuration represents the amount of resources used by a single computing instance; performance statistics for computing instances under different resource configurations are essentially performance statistics of computing instances when using different amounts of resources. Performance statistics can refer to time consumption data, for example. Based on the performance statistics, time consumption models under different resource configurations can be constructed.

[0082] Before the model reinforcement training process begins, for at least one computation stage (e.g., the generation stage), the optimal resource configuration for the computation instances in that stage can be determined using a time-consuming model based on the initial resource volume and request scale of that stage, in order to initialize the deployment of the at least one computation stage. In other words, by utilizing the time-consuming model of computation instances under different resource configurations, the resource configuration that minimizes the total time consumption for a given request scale and resource volume can be determined as the optimal resource configuration for a single computation instance. Based on the optimal resource configuration, the initial deployment of the computation stage can be achieved. For example, once the optimal resource configuration for a single computation instance in a certain computation stage is determined, the corresponding computation instances can be deployed according to the optimal resource configuration, using the resources allocated to that computation stage, to complete the initial deployment of that computation stage.

[0083] It should be understood that when scaling up the second computing phase by implementing the resource scheduling scheme of this disclosure, the second computing instance can also be deployed using the resources allocated to the second computing phase, based on the optimal resource configuration of the second computing instance determined in advance for the second computing phase, in order to complete the scaling up of the second computing phase.

[0084] In other words, the amount of resources occupied by each computing instance (e.g., the first computing instance, the second computing instance) in the initial state of this disclosure can be the optimal resource configuration of the corresponding computing instance that has been predetermined.

[0085] For example, an inference engine instance during the generation phase can be deployed on either 4 GPUs or 8 GPUs. While 8 GPUs might be faster, the number of instances that can be deployed is limited by the available GPUs. Conversely, 4 GPUs might be slower, but allow for a larger number of deployed instances. Therefore, a time-consuming model can be used to determine which resource configuration is better, thus identifying the optimal resource allocation for a single inference engine instance.

[0086] In some embodiments, each training round includes multiple computational phases, including a generation phase, an inference phase, and a training phase. Considering the cost of resource scheduling, this specification proposes allocating a fixed amount of resources to the inference phase, while in each training round, the generation phase is scaled down and the training phase is scaled up. That is, in each training round, only the generation and training phases participate in scaling up and down, while the inference phase remains unchanged. Specifically, the first computational phase mentioned above includes the generation phase, and the second computational phase includes the training phase.

[0087] In other words, at the beginning of each training round, a fixed first portion of resources from the resource pool can be allocated to the inference phase, and a second portion of resources, excluding the first portion, can be allocated to the generation phase. The amount of resources in the second portion should be greater than (e.g., much greater than) the amount of resources in the first portion.

[0088] In some further embodiments, the training samples for each training epoch can be evenly divided into multiple smaller global batches during the training phase. For each global batch, the training engine performs one forward computation and one backpropagation, and uses the optimizer to update the parameters within the training engine before training the next global batch.

[0089] Before the first global batch begins, steps S220 and S230 can be executed in response to the detection that the number of remaining requests in the generation phase is below a first threshold (e.g., below 50%), to scale down the generation phase and scale up the training phase. Before the start of each global batch after the first global batch, steps S220 and S230 can be executed directly based on the currently detected number of remaining requests in the generation phase.

[0090] Therefore, before each global batch begins, the generation phase can be scaled down once and the training phase can be scaled up once. Thus, within a training round, the generation phase continuously scales down and the training phase gradually scales up over time, maximizing resource allocation in the training phase while meeting the generation phase rate, thereby improving resource utilization.

[0091] Figure 4 A schematic diagram of the architecture of a scheduling system according to some embodiments of the present disclosure is shown.

[0092] See Figure 4 The scheduling system includes an execution plane and a scheduling plane. The scheduling system can also be referred to as an automatic scaling system for model reinforcement training or a model reinforcement training system. In some embodiments, the execution plane can be deployed on the GPU, and the scheduling plane can be deployed on the CPU. The deployment of the execution plane and the scheduling plane can be configured according to actual conditions, and this disclosure does not impose any restrictions on this.

[0093] The execution plane includes the runtime. The runtime comprises several executable modules and their data dependencies. For example, training a large model using GRPO (such as a large model specifically designed for mathematical problems) includes modules for generation, inference, and training. Each module needs to reserve an interface for the scheduling plane. This functionality includes two parts: one part returns information needed by the scheduling plane, such as runtime status, available scheduling time, and resource limits; the other part includes unloading and migration actions for the scheduling plane to invoke. Runtime status represents the number of requests currently running (i.e., the number of remaining requests). Available scheduling time represents the time during which the module can schedule resources. Resource limits can include two aspects: the available free resources the module can provide, and the acceptable resources for the module (e.g., acceptable resources must be an integer multiple of the number of GPUs used by a single computing instance). Figure 4 The runtime shown is an abstract concept. Figure 4 The interaction between the runtime and the scheduling plane is the interaction between the modules in the runtime and the scheduling plane.

[0094] The scheduling plane includes a performance analyzer, a monitor (also called a monitor), a scheduler, and a controller.

[0095] The performance analyzer can obtain performance statistics under different parallel configurations before the actual runtime, providing the optimal configuration for different modules under given computing resources and requests, and establishing a time-consuming model. The time-consuming model is a formula-based model established based on the request size, model size, computational load, and hardware metrics, and fitted using actual runtime. After its establishment, the current system's state metrics can be used as input to predict the execution time of each module. For more information on the time-consuming model and optimal resource configuration, please refer to the relevant descriptions above.

[0096] A monitor is used to collect and analyze runtime data and monitor runtime status. The monitor is configured to monitor the runtime status of at least one first computation stage among multiple computation stages to obtain first runtime status information, which includes a first number of remaining requests.

[0097] The scheduler obtains real-time runtime status from the monitor and, based on the time consumption model provided by the performance analyzer, makes scheduling decisions at appropriate times according to a specific algorithm, reallocating resource configurations and parallel strategies for each module. The algorithm aims to maximize throughput and device utilization while minimizing execution time. The scheduler is configured to determine the scalable resources for the first computation phase based on the first remaining request count, the target batch size of each first computation instance in the first computation phase, and the resource consumption of each first computation instance.

[0098] The controller, based on the scheduler's decisions, issues specific instructions to the runtime (i.e., to the corresponding modules within the runtime) to execute scheduling decisions. The controller is configured to migrate at least some scalable resources to at least one second computing stage among multiple computing stages.

[0099] To meet the needs of system scheduling, each module needs to implement certain interfaces for the monitor and controller to call. First, it needs to provide an interface for returning information about its running status, which the monitor can call to obtain and report back to the scheduler as a basis for decision-making. Second, it needs to implement scaling functionality and provide the interface to the controller. Scaling functionality is the core component of the entire scheduling process. In some exemplary embodiments, this disclosure designs and implements scaling schemes for the inference engine and the Megatron training engine, respectively.

[0100] The scheduling algorithm is the core part of the scheduler. It takes the running status of each component obtained by the monitor and the time consumption model established by the performance analyzer as input, and aims to maximize resource utilization and throughput. It makes decisions on scaling up or down each module, and the controller executes these decisions at runtime.

[0101] The following is combined Figure 4 An example of a typical scheduling algorithm will be given.

[0102] In each training round of GRPO, after the generation and inference phases, all input sequences are divided into several smaller batches (global batches). For each global batch, the training engine performs a forward computation and a backward propagation, and uses the optimizer to update the parameters within the training engine before training the next global batch.

[0103] The scheduling algorithm can scale up or down before and after the start of each global batch training. In reinforcement learning training, since the generation phase takes the longest time and is the first phase of each training round, at the very beginning, apart from the GPUs fixed for the inference phase, the scheduler will allocate all other resources to the generation phase first, triggering the scaling up of the generation phase, while the training phase does not occupy any GPUs temporarily.

[0104] The monitor continuously acquires information such as the number of remaining requests during the generation phase and the distribution of requests across several LLM inference engines (corresponding to the first compute instance mentioned above) at regular intervals. Once the number of remaining requests drops below a pre-set threshold (e.g., 50%), the scheduling algorithm determines the number of GPUs that need to be retained during the generation phase based on the number of requests still running and the time consumption model, thus determining the number of GPUs that can be released. See the relevant description above for the specific calculation method.

[0105] The scheduler then decides how many GPUs to allocate to the training phase. Because there's a constraint between parallel configuration (i.e., the number of training engine instances) and batch size—the number of training engine instances must be divisible by the batch size—the size of the training phase must be set within a pre-defined set of values ​​(for example, if a single training engine instance uses 4 GPUs, the value must be a multiple of 4, and the number of training engine instances must be divisible by the batch size). The scheduler will select the maximum value within this set that is not greater than the number of GPUs that can be released as the scheduling result.

[0106] After the above two stages, the scheduler determines the actual number of GPUs to be released during the generation stage (that is, the number of GPUs to be allocated to the training stage). The scheduler then passes this decision to the controller to scale down the generation stage and scale up the training stage.

[0107] After the first global batch is trained during the training phase, the scheduler will repeat the above process until all global batches within the round are trained. The difference is that before subsequent global batches begin, scaling up or down can be performed directly based on the currently monitored number of remaining requests, without needing to determine if the number of remaining requests has fallen below a threshold.

[0108] Within a training round, the generation phase shrinks continuously over time while the training phase expands gradually. This maximizes resource allocation in other phases while satisfying the generation phase's speed requirements, thereby improving resource utilization.

[0109] In summary, this disclosure introduces a dynamic monitoring and scheduling mechanism during the reinforcement learning training process, and implements a dynamic scaling-up and scaling-down mechanism for each module. This allows modules with low resource utilization to be scaled down at appropriate times, freeing up resources for other stages, thereby improving the overall resource utilization and throughput of the system.

[0110] This disclosure also proposes a resource scheduling device for a model reinforcement training process, wherein the model reinforcement training process includes one or more training rounds, each training round including multiple computation stages, comprising: a monitor configured to monitor the running status of at least one first computation stage among the multiple computation stages, and obtain first running status information, the first running status information including a first remaining number of requests; a scheduler configured to determine scalable resources for the first computation stage based on the first remaining number of requests, the target batch size of each first computation instance in the first computation stage, and the amount of resources occupied by each first computation instance; and a controller configured to migrate at least a portion of the scalable resources to at least one second computation stage among the multiple computation stages. Other embodiments of the monitor, scheduler, and controller are described in the foregoing embodiments and will not be repeated here.

[0111] This disclosure also proposes another resource scheduling apparatus for a model reinforcement training process, including a memory; and a processor coupled to the memory, the processor being configured to execute the resource scheduling method for a model reinforcement training process in any embodiment of this disclosure based on instructions stored in the memory.

[0112] This disclosure also proposes a model reinforcement training system. The model reinforcement training process includes one or more training rounds, each training round including multiple computational stages. The system includes a scheduling device and an execution device. The scheduling device corresponds to the scheduling plane described above. The execution device corresponds to the execution plane described above.

[0113] The execution device includes multiple computing modules, each corresponding to a computing stage.

[0114] The scheduling device monitors the operating status of at least one first computing module among multiple computing modules, obtaining first operating status information of the first computing module, including a first remaining number of requests. Based on the first remaining number of requests, the target batch size of each first computing instance in the first computing phase, and the amount of resources occupied by each first computing instance, the scheduling device determines the scalable resources for the first computing phase. The scheduling device controls the execution device to migrate at least a portion of the scalable resources to at least one second computing module among the multiple computing modules. For details regarding the operations that the scheduling device and the execution device can perform, as well as the related details, please refer to the relevant descriptions above.

[0115] Furthermore, the method according to this disclosure can also be implemented as a computer program or computer program product, which includes computer program code instructions for performing the steps defined in the above-described method of this disclosure.

[0116] Alternatively, this disclosure may be implemented as a non-transitory machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium) storing executable code (or computer program, or computer instruction code) that, when executed by a processor of an electronic device (or computing device, server, etc.), causes the processor to perform the steps of the method described above according to this disclosure.

[0117] Those skilled in the art will also understand that the various exemplary logic blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein can be implemented as electronic hardware, computer software, or a combination of both.

[0118] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems and methods according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0119] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A resource scheduling method for model reinforcement training, wherein, The model reinforcement training process includes one or more training rounds, each training round including multiple computational stages, and the method includes: Monitor the running status of at least one first computing stage among the plurality of computing stages to obtain first running status information, the first running status information including a first remaining number of requests; Based on the first remaining request quantity, the target batch size of each first computing instance in the first computing phase, and the amount of resources occupied by each first computing instance, the scalable resources of the first computing phase are determined. Migrate at least some scalable resources to at least one of the multiple computing stages in a second computing stage. The target batch size for each first computing instance refers to the batch size when the throughput of the first computing instance is higher than a preset level. Based on the first remaining request count, the target batch size of each first computing instance in the first computing phase, and the amount of resources occupied by each first computing instance, the scalable resources for the first computing phase are determined, including: determining the minimum set of first computing instances to be retained based on the first remaining request count, the target batch size of each first computing instance, and the amount of resources occupied by each first computing instance, wherein the sum of the target batch sizes of all first computing instances in the first computing instance set is greater than or equal to the first remaining request count; and determining scalable resources based on the first computing instance set, wherein the scalable resources include the resources remaining after deducting the resources occupied by all first computing instances in the first computing phase from the resources occupied by all first computing instances in the first computing phase.

2. The method according to claim 1, wherein, The first running status information also includes the distribution of the number of remaining requests across different first computing instances, determining the minimum set of first computing instances that need to be retained, including: In descending order of the number of remaining requests, first computing instances are selected one by one from all the current first computing instances in the first computing phase until the sum of the target batch sizes of the cumulative selected first computing instances is greater than or equal to the number of remaining requests. Then the selection stops. The set of cumulatively selected first computing instances is the minimum set of first computing instances that need to be retained.

3. The method according to claim 2, wherein, Migrating at least some scalable resources to at least one second computing stage among the plurality of computing stages includes: Unload at least a portion of the first computing instances outside the first set of computing instances; The resources occupied by at least a portion of the first computing instance that have been unloaded from the scalable resources are allocated to the second computing phase.

4. The method according to any one of claims 1 to 3, wherein, Migrating at least some scalable resources to at least one second computing stage among the plurality of computing stages includes: Obtain the second remaining request count in the second calculation phase; Based on the second remaining request quantity and the amount of resources required by a single second computing instance in the second computing phase, determine the resource constraints when expanding the second computing phase; Allocate the resources that meet the resource constraints from the scalable resources to the second computing stage.

5. The method according to any one of claims 1 to 3, further comprising: The first time consumption model is used to predict the time consumption of the first computing instance under different batch sizes; Based on the predicted time consumption, the throughput index of the first computing instance under different batch sizes is calculated. The optimal batch size for throughput is determined as the target batch size.

6. The method according to any one of claims 1 to 3, further comprising: Pre-obtain performance statistics of computing instances in at least one computing stage under different resource configurations, wherein the resource configuration represents the amount of resources occupied by a single computing instance, and the performance statistics include at least time consumption data; Based on the aforementioned performance statistics, a time consumption model under different resource configurations is constructed; as well as Before the model reinforcement training process begins, for the at least one computing stage, based on the initial resource amount and request scale of the computing stage, the optimal resource configuration of the computing instance of the computing stage is determined using the time consumption model to initialize the deployment of the at least one computing stage.

7. The method according to any one of claims 1 to 3, wherein, The plurality of computational stages include a generation stage, an inference stage, and a training stage; the first computational stage includes the generation stage; the second computational stage includes the training stage; and the method further includes: At the beginning of each training round, a fixed first portion of resources in the resource pool is allocated to the inference phase, and a second portion of resources in the resource pool other than the first portion are allocated to the generation phase, wherein the amount of the second portion of resources is greater than that of the first portion of resources.

8. The method according to claim 7, wherein, The training samples for each training round are divided into multiple global batches during the training phase. Before the first global batch begins, in response to the first remaining request quantity being lower than a first threshold, the steps of determining the scalable resources of the first computing stage and migrating at least a portion of the scalable resources to at least one second computing stage among the plurality of computing stages are executed. Before each subsequent global batch begins, the steps of determining the scalable resources of the first computing stage and migrating at least a portion of the scalable resources to at least one second computing stage among the plurality of computing stages are performed.

9. A resource scheduling device for model reinforcement training, wherein, The model reinforcement training process includes one or more training rounds, and each training round includes multiple computational stages, including: A monitor is configured to monitor the running status of at least one first computing stage among the plurality of computing stages, and obtain first running status information, the first running status information including a first number of remaining requests; The scheduler is configured to determine the scalable resources of the first computing phase based on the first remaining number of requests, the target batch size of each first computing instance in the first computing phase, and the amount of resources occupied by each first computing instance. The controller is configured to migrate at least a portion of the scalable resources to at least one second computing stage among the plurality of computing stages. The target batch size for each first computing instance refers to the batch size when the throughput of the first computing instance is higher than a preset level. The scheduler determines the minimum set of first computing instances to be retained based on the first remaining request count, the target batch size of each first computing instance, and the amount of resources occupied by each first computing instance. The sum of the target batch sizes of all first computing instances in the first computing instance set is greater than or equal to the first remaining request count. Based on the first computing instance set, the scheduler determines scalable resources, which include the resources remaining after deducting the resources occupied by all first computing instances in the first computing phase from the resources occupied by all first computing instances in the first computing phase.

10. A model reinforcement training system, wherein, The model reinforcement training process includes one or more training rounds, each training round including multiple computational stages. The system includes a resource scheduling device and an execution device for the model reinforcement training process. The execution device includes multiple computing modules, each computing module corresponding to one computing stage. The resource scheduling device is configured as follows: By monitoring the operating status of at least one first computing module among the plurality of computing modules, first operating status information of the first computing module is obtained, the first operating status information including a first remaining number of requests. Based on the first remaining request count, the target batch size of each first computing instance in the first computing module, and the resource consumption of each first computing instance, the scalable resources of the first computing module are determined. The actuator is controlled to migrate at least a portion of the scalable resources to at least one second computing module among the plurality of computing modules. Wherein, the target batch size of each first computing instance refers to the batch size when the throughput index of the first computing instance is higher than a preset level. The resource scheduling device determines the minimum set of first computing instances that need to be retained based on the first remaining request quantity, the target batch size of each first computing instance, and the amount of resources occupied by each first computing instance. The sum of the target batch sizes of all first computing instances in the first computing instance set is greater than or equal to the first remaining request quantity. The resource scheduling device determines the scalable resources based on the first computing instance set. The scalable resources include the resources remaining after deducting the resources occupied by all first computing instances in the first computing module from the resources occupied by all first computing instances in the first computing instance set.

11. A non-transitory machine-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to perform the method as claimed in any one of claims 1 to 8.

Images