Reinforcement learning-oriented elastic computing system and workflow arrangement method

Abstract

The present application relates to a reinforcement learning oriented elastic computing system for orchestrating and executing reinforcement learning workflows on a cluster of heterogeneous computing devices. The system comprises multiple work units, each encapsulating a reinforcement learning component, supporting resource on-demand offloading and providing communication primitive interfaces; a workflow executor receiving a macro logic flow describing data interaction among work units defined in imperative programming; a scheduler converting the macro logic flow into micro execution flow, including dynamic granularity distribution through elastic pipeline mechanism in spatial dimension and resource time multiplexing of work units through a context switching mechanism in temporal dimension; an adaptive communication layer perceiving physical placement information of work units, managing connection life cycle and automatically selecting communication backends. The system realizes efficient orchestration and execution of highly heterogeneous reinforcement learning workflows, significantly improving resource utilization and system execution efficiency.

Description

Technical Field

[0001] This invention relates to the field of design and scheduling execution strategies for reinforcement learning systems, and in particular to a flexible computing system and workflow orchestration method for reinforcement learning. Background Technology

[0002] The development of large-scale language models has reached a bottleneck, with the benefits of simply increasing model size gradually diminishing. To achieve breakthroughs in intelligence, reinforcement learning has become a key paradigm, widely applied in areas such as human feedback-based reinforcement learning, group-based policy optimization, and embodied agent training, to align with human preferences, enhance reasoning abilities, and enable autonomous interaction with complex environments. It is predicted that the computational resources consumed by reinforcement learning workloads will soon surpass those of large-scale model pre-training, making training efficiency a core challenge in system design.

[0003] However, large-scale reinforcement learning training for diverse scenarios such as reasoning, intelligent agents, and embodied interaction faces significant efficiency bottlenecks. This is because these workloads integrate highly heterogeneous components, such as large model generation / inference / training, reward models, value function models, tool invocation environments, and embodied simulators. Each component has distinct operating characteristics and resource requirements: large model training needs to save gradients and optimizer states, resulting in significantly higher GPU memory usage than generation tasks; while large model generation tasks, due to dynamically changing response lengths, are prone to low computational resource utilization. Further complicating matters, different components require different parallelization strategies—large model training supports tensor parallelism, pipelined parallelism, and other methods, while components like embodied simulators require a mix of CPU physics simulation and GPU graphics rendering, typically only scalable through instance duplication.

[0004] The existing single execution mechanism of reinforcement learning training systems is ill-suited to this inherent heterogeneity, leading to significant efficiency losses. Currently, there are two main execution modes: the first is a cooperative execution mode, where different components sequentially occupy the accelerator; this mode suffers from significant resource idleness due to the long-tail effect of generated tasks. The second is a decoupled mode, which achieves concurrent execution of components through pipeline parallelism. While this alleviates the long-tail problem, it introduces imbalances in memory and computational load between stages. Neither mode is a universally applicable solution; many practical scenarios require hybrid scheduling to achieve optimal efficiency.

[0005] More fundamentally, existing systems suffer from insufficient architectural flexibility. They typically employ a graph-based declarative programming paradigm, sacrificing control flow flexibility, debuggability, and system transparency in pursuit of optimization. This makes it difficult for developers to flexibly switch execution modes to adapt to different workload characteristics while maintaining programming consistency.

[0006] Therefore, there is an urgent need in this field for a new type of reinforcement learning computing system and method that can fundamentally solve the above-mentioned technical contradictions, namely, achieving efficient resource scheduling and utilization while being compatible with highly heterogeneous component workflows, and providing developers with a flexible and intuitive programming experience.

[0007] It should be noted that the information disclosed in the background section above is only for understanding the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0008] The technical problem this application aims to solve is "how to address the issues of low computational resource utilization and low execution efficiency in existing reinforcement learning training systems when processing workflows that integrate highly heterogeneous components such as large language models, reward models, and simulators".

[0009] The technical solution adopted in this application to solve the above-mentioned technical problems is as follows.

[0010] This application provides an elastic computing system for reinforcement learning, used to orchestrate and execute reinforcement learning workflows on a heterogeneous computing device cluster. The system includes: multiple work units, each encapsulating a reinforcement learning component; each work unit supports on-demand resource unloading and provides a communication primitive interface; a workflow executor for receiving a macro-logic flow defined through imperative programming, describing data interactions between the multiple work units; a scheduler, communicatively connected to the workflow executor, for converting the macro-logic flow into micro-execution flows; and an adaptive communication layer for sensing the physical placement information of the multiple work units, managing the connection lifecycle between the multiple work units, and automatically selecting a communication backend based on the physical placement information to support data transmission between the multiple work units. The scheduler's conversion of the macro-logic flow into micro-execution flows includes: dynamically distributing work units at granularity in the spatial dimension through an elastic pipeline mechanism, and performing time-division multiplexing of resources for work units in the temporal dimension through a context switching mechanism.

[0011] In some embodiments, multiple work units inherit from the same base class, which provides send and receive primitives for communication between the multiple work units, as well as load and unload functions for resource management.

[0012] In some embodiments, the scheduler further includes an analyzer module for establishing an execution time model for a work unit; the scheduler is configured to dynamically generate an optimal execution plan by recursively dividing the workflow graph based on the execution time model, thereby evaluating the cost of using spatial scheduling and time scheduling strategies for each subgraph generated after the workflow graph is divided; the analyzer module uses a polynomial extrapolation method to establish an execution time model for a work unit under different numbers of computing devices.

[0013] In some embodiments, the elastic computing system for reinforcement learning also includes a data channel, which is a first-in-first-out queue used to decouple the control flow and data flow among multiple work units that have data dependencies on each other.

[0014] In some embodiments, time-division multiplexing of resources for work units through a context switching mechanism includes: restricting concurrent access to multiple work units with data flow dependencies on the same computing device in a heterogeneous computing device cluster through a distributed device lock; wherein, a work unit needs to acquire the distributed device lock before accessing resources; the acquisition priority of the distributed device lock is defined by the data dependency information between work units in the data channel.

[0015] In some embodiments, the adaptive communication layer is configured to support multiple communication backends, including a collection communication library backend for inter-GPU communication, an inter-process direct memory access backend for intra-GPU communication, and a distributed communication framework backend for inter-CPU communication, and can automatically select the optimal backend based on physical placement information.

[0016] In some embodiments, the flexible pipeline mechanism achieves dynamic granularity distribution of work units by dynamically adjusting the data processing granularity input to the work unit; wherein, the work unit is configured to process data at different batch sizes, and when the output data of the current work unit reaches a configurable batch size, the batch data is forwarded to the next work unit, so that the next work unit can start early or delay processing at a granularity corresponding to the batch size.

[0017] In some embodiments, the elastic computing system for reinforcement learning further includes a runtime monitor for continuously recording the actual execution characteristics of the work units; when the deviation between the actual execution characteristics and the estimated execution characteristics on which the scheduler is based exceeds a threshold, the scheduler is triggered to reschedule in order to generate an updated micro-execution flow.

[0018] In some embodiments, a reinforcement learning workflow orchestration method is also provided, running on a heterogeneous computing device cluster. The reinforcement learning workflow orchestration method includes: defining a macro logic flow describing data interaction between multiple work units through imperative programming, wherein each work unit encapsulates a reinforcement learning component and supports resource unloading on demand; converting the macro logic flow into a micro execution flow; sensing the physical placement information of multiple work units, and automatically selecting a communication backend to manage data transmission based on the physical placement information; wherein, converting the macro logic flow into a micro execution flow includes: dynamically distributing work units at a granular level in the spatial dimension through an elastic pipeline mechanism, and performing time-division multiplexing of resources for work units in the temporal dimension through a context switching mechanism.

[0019] In some embodiments, a computer-readable storage medium is also provided, on which a computer program / instruction is stored, which, when executed by a processor, implements the reinforcement learning workflow orchestration method of the present invention.

[0020] The present invention has the following beneficial effects: This invention, through the technical feature of "multiple working units, each encapsulating a reinforcement learning component, with each working unit supporting on-demand resource offloading and providing communication primitive interfaces," transforms highly heterogeneous reinforcement learning components into schedulable units with unified resource management and communication interfaces, providing fundamental support for unified management and flexible scheduling of heterogeneous components. Through the technical feature of "a workflow executor for receiving macro-logic flows described by imperative programming, depicting data interactions between multiple working units," developers can flexibly and intuitively define the macro-logic of workflows using imperative programming, overcoming the shortcomings of existing declarative programming paradigms in control flow flexibility. Through the technical feature of "a scheduler, communicating with the workflow executor, for converting macro-logic flows into micro-execution flows," the invention achieves the automatic conversion of developer-defined macro-logic flows into micro-execution flows that the system can efficiently execute. Key functions provide a core conversion mechanism for deep system-level optimization; through the technical feature of "adaptive communication layer, used to perceive the physical placement information of multiple work units, manage the connection lifecycle between multiple work units, and automatically select communication backends for data transmission based on physical placement information", it ensures high efficiency and low latency of data transmission between work units in heterogeneous clusters by perceiving the physical location of work units and adaptively selecting the optimal communication backend; through the technical feature of "dynamic granular distribution of work units in the spatial dimension through elastic pipeline mechanism", it realizes fine-grained spatial scheduling of work units, which can effectively adapt to the computing characteristics of different components and alleviate the problem of load imbalance; through the technical feature of "time-division multiplexing of resources for work units through context switching mechanism in the time dimension", it allows multiple work units to safely and automatically share the same hardware resources in time-division, significantly reducing resource idleness.

[0021] In summary, this invention, through the synergistic interaction of the aforementioned technical features, forms an organic whole, jointly achieving efficient orchestration and execution of highly heterogeneous reinforcement learning workflows. This ultimately leads to a significant improvement in computational resource utilization and system execution efficiency. Specifically: First, this invention standardizes and encapsulates reinforcement learning components through work unit abstraction; the workflow executor receives macro-logic flows defined by imperative programming, preserving flexibility on the development side; subsequently, the core scheduler converts the macro-logic flows into micro-execution flows. This conversion process is specifically implemented through a flexible pipeline mechanism and a context switching mechanism, respectively, performing deep optimization scheduling in the spatial and temporal dimensions. The former addresses load balancing, while the latter addresses resource idleness; simultaneously, an adaptive communication layer ensures efficient data transmission throughout the scheduling and execution process. Ultimately, these technical features form an organic whole, jointly achieving full utilization of heterogeneous resources, thus systematically solving the technical problem of "how to address the low computational resource utilization and low execution efficiency of existing reinforcement learning training systems when processing workflows that integrate highly heterogeneous components such as large language models, reward models, and simulators."

[0022] Other beneficial effects of the present invention will be further described below. Attached Figure Description

[0023] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is a diagram of the reinforcement learning elastic computing system architecture of the present invention; Figure 2 This is a schematic diagram of the macro-microfluidic conversion execution logic of the present invention; Figure 3 This is a schematic diagram of the time and air conditioning mode of the present invention; Figure 4 This is a comparison chart of the throughput of the system of this invention and the veRL baseline on large language model reinforcement learning tasks; Figure 5 This is a comparison chart of the throughput of the system of this invention and the Slime baseline on large language model reinforcement learning tasks; Figure 6 This is a comparison chart of the throughput of the system of the present invention and the reinforcement learning task of a large language model with multiple baselines; Figure 7 This is a comparison chart of the throughput of the embodied intelligence reinforcement learning task in different execution modes of the system of the present invention. Detailed Implementation

[0024] The embodiments of the present invention will be described in detail below. It should be emphasized that the following description is merely exemplary and not intended to limit the scope and application of the present invention.

[0025] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of the present invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0026] This invention addresses the fundamental technical problems of existing reinforcement learning systems when handling complex heterogeneous workflows—namely, insufficient control flow flexibility due to the use of declarative programming paradigms, and low hardware utilization caused by the inability of static scheduling strategies to adapt to the diverse resource requirements and operational characteristics of heterogeneous components—and proposes a groundbreaking solution. At its root, these technical problems stem from the failure of existing technologies to recognize that thoroughly decoupling the high-level logical definition of the workflow (macro-logic flow) from its underlying physical execution scheduling (micro-execution flow) is key to achieving efficient and flexible resource scheduling. Based on this profound insight, this invention creatively proposes a core design paradigm of "macro-micro flow transformation."

[0027] This paradigm achieves architectural isolation between code logic and physical execution by encapsulating reinforcement learning components into work units that support on-demand resource offloading and unified communication primitives. Furthermore, the system's scheduler analyzes the runtime characteristics of each work unit and, based on recursive graph partitioning and cost assessment models, automatically transforms the macro-logic flow defined by developers through imperative programming into micro-execution flows optimally orchestrated in the spatiotemporal dimensions. This transformation is specifically achieved through the following mechanisms: in the spatial dimension, a flexible pipeline mechanism is used for dynamic granular distribution, allowing work units to execute at the most appropriate data batch size; in the temporal dimension, a context switching mechanism is introduced to achieve automatic and secure time-division multiplexing of shared computing devices. In addition, the system intelligently manages connection lifecycles and heterogeneous data transmission through a location-aware adaptive communication layer, thereby constructing a flexible and efficient execution environment on a distributed heterogeneous computing power cluster.

[0028] It is understood that this invention achieves a balance between programming flexibility, execution efficiency, and resource utilization in the field of reinforcement learning systems. It allows developers to express complex logic through intuitive procedural programming, while the system automatically performs extreme performance optimization at the underlying level. Ultimately, it effectively solves the long-standing technical challenge of resource contention and idleness in large-scale heterogeneous clusters, significantly improving training efficiency and computing power utilization.

[0029] The following will further describe specific embodiments of the present invention. These embodiments are merely illustrative and do not mean that the present invention is limited to the following examples.

[0030] In some embodiments, to achieve an efficient, flexible, and intuitive elastic computing system for reinforcement learning, this invention proposes a programming paradigm based on macro-microflow transformation. In this paradigm, developers define the logical communication flow between reinforcement learning components (hereinafter referred to as "macro-logic flow") at a coarse-grained level through imperative programming; the system then automatically converts this macro-logic flow into a fine-grained execution flow (hereinafter referred to as "micro-execution flow"). Essentially, macro-microflow transformation decouples programmable code logic from the physical execution and scheduling processes of reinforcement learning components, thereby significantly reducing programming complexity while maximizing execution efficiency.

[0031] In some embodiments, Figure 1 The system architecture for implementing macro-to-microfluidic transformation is shown. The system provides users with an easy-to-use procedural programming interface for building reinforcement learning workflows that describe data communication and interaction between components. In the system, reinforcement learning components are encapsulated as "work units," each implementing the core logic of that component. Work units support free communication with each other and have a resource offloading mechanism to achieve time-division multiplexing of hardware resources. This work unit abstraction allows the system to retain sufficient scheduling flexibility in both spatial and temporal dimensions while adhering to the procedural workflow: spatial scheduling allocates work units to specific accelerators, temporal scheduling determines their execution periods, and temporal scheduling controls the granularity of pipeline execution.

[0032] In some embodiments, to generate the optimal micro-execution flow across the aforementioned scheduling dimensions, the core of the scheduler module lies in a scheduling strategy based on an execution time model. This strategy analyzes the runtime characteristics of work units to search for the optimal execution mode for each work unit. Based on the determined execution mode, the controller assigns work units to corresponding accelerators, manages connections between components, and orchestrates the execution flow by dispatching function calls. This invention designs two mechanisms: flexible pipelines and context switching, to respectively implement spatial and temporal orchestration of work units. Furthermore, adaptive communication tools such as point-to-point communication and data channels serve as the data plane, supporting scalable work unit interaction.

[0033] In some embodiments, at the distributed execution and underlying cluster control levels, the entire system deeply integrates Ray (A Distributed Framework for Emerging AI Applications), a high-performance open-source framework designed specifically for large-scale distributed computing and machine learning applications. Addressing the difficulty of cross-node collaboration between modules in the original system, the system highly encapsulates Ray's Actor model, making it serve as the distributed cluster management and communication infrastructure for the entire system.

[0034] Specifically, this integration method and its interaction with various modules of the system are reflected in the following three aspects: Unified cluster management and dynamic process startup: The system utilizes Ray to shield the topology differences of the underlying physical nodes, constructing a global logical resource pool. The master node can dynamically allocate resources, start and destroy worker processes on any target node in the cluster through encapsulated interfaces.

[0035] Asynchronous remote calls between modules: Core components within the system (such as the analyzer and scheduler) are all packaged as distributed RayActors with independent state. These modules do not need to use cumbersome network socket programming, but instead interact seamlessly using Ray's remote call mechanism. For example, the scheduler can issue execution pipelines to work units across nodes, and the work units asynchronously return their execution status.

[0036] Collaborative Reinforcement Learning Component Behavior: Addressing the need for large-scale distributed environment interaction in reinforcement learning, the system encapsulates and deploys key RL components (such as the environment simulator, Rollout engine, and reward signals) as independent RayActors. Under this architecture, the system can directly manipulate and control the behavior of these reinforcement learning components. For example, it can trigger state exploration (Rollout) in parallel across cluster nodes to collect environmental feedback, asynchronously transmit reward signals across nodes, and centrally or distributedly trigger model gradient updates and policy distribution.

[0037] Through this deep integration, the system has completely opened up the communication links between scheduling planning, performance analysis and reinforcement learning components, breaking through the performance bottleneck of a single machine and realizing high-concurrency scheduling and flexible control under large-scale clusters.

[0038] In some embodiments, the system is designed to achieve high efficiency by maximizing system flexibility, which is also the guiding principle for its programming interface design. Unlike traditional graph-based declarative programming, which often sacrifices control flow flexibility, debuggability, and system transparency for optimization, this system adopts a procedural programming paradigm, allowing developers to flexibly express workflows in an imperative manner. Example workflows based on this interface are shown in Tables 1 and 2.

[0039] Table 1 Table 2 This reinforcement learning program consists of two parts: (1) Define the working unit program for each reinforcement learning component logic; (2) By calling the core functions of the work unit and defining its interaction logic, the workflow executor of the overall workflow is arranged.

[0040] In some embodiments, Table 1 shows a typical work unit implemented on the system. The base Worker class provides communication primitives such as send and recv for communication between work units. All work units inherit from this base class and automatically acquire communication capabilities, which is also the foundation for higher-level communication facilities such as data channels. To manage constrained device resources, each work unit implements resource management functions (onload and offload) to perform resource acquisition and release.

[0041] In some embodiments, after implementing the work unit, developers can refer to Table 2 to combine reinforcement learning workflows. First, the executor starts the work unit on the node and device cluster in SPMD (Single Program Multiple Data) mode. The scheduler module determines the placement of the work unit before it starts, or the user can specify it manually. All processes within the same work unit are managed uniformly through the system's Worker Group abstraction (e.g., rollout_group). When a Worker Group is called, it automatically proxies the common functions defined in the work unit class and dispatches them to all or some of the work unit processes. Worker Group function calls are inherently asynchronous and return result handles; its wait primitive provides synchronization barriers, supporting computation at specific data granularities. For example, in GRPO training, rollout can be executed on a single request basis, but the stimulus normalization process must aggregate all group requests; in this case, the pipeline needs to pause at this step until normalization is complete. The core infrastructure connecting the data flow between distributed work unit groups is the data channel, which decouples the control flow and data flow of dependent components, thereby improving programming flexibility and providing ample optimization space.

[0042] In some embodiments, the programming interface shown in Table 2 provides a flow-based programming model to describe the high-level, logical control and data flows between work units. After acquiring the logical flow, the system follows a macro-microflow transformation paradigm to convert the logical flow into a concrete execution flow, i.e., determining the running location (spatial dimension) and running time (temporal dimension) of the work unit. The following focuses on two core enabling mechanisms for implementing macro-microflow transformation and flexible scheduling: elastic pipelines and context switching.

[0043] Space scheduling is achieved through flexible pipelines: In some embodiments, for spatial scheduling, work units can be executed in a pipelined manner on different numbers of accelerators or devices. To maximize pipeline flexibility, the system introduces a flexible pipeline, enabling work units to process data at different granularities within given device resource constraints. The flexible pipeline is based on the following technical insight: in reinforcement learning training and agent scenarios, most work units follow the SPMD pattern, allowing execution at different batch sizes. For example, a large model service engine can process one or more cues at a time, and the inference process also supports single-batch or multi-batch execution. This flexibility allows the system's execution flow manager to achieve flexible pipelined processing of tasks through dynamic data granularity: once a data batch of the configured size is ready, output data can be forwarded, allowing downstream work units to start earlier in smaller batches or process later in larger batches. It is worth noting that the scheduling space is influenced by the computational semantics within each work unit; for example, training work units simultaneously have the concepts of micro-batches (defining forward / backward units) and global batches (determining when model updates occur).

[0044] Time scheduling is achieved through automatic context switching: In some embodiments, in addition to spatial scheduling, the system also supports natural time-division multiplexing of devices through automatic context switching. Context switching enables work units that cannot coexist on the same accelerator due to device resource limitations to share devices through sequential execution. In this system, this is achieved through distributed device locks (i.e., device_lock shown in Table 1) in the data channel facility. This lock, as a primitive, is used to restrict concurrent access to resources by multiple work units with data flow dependencies on the same device. Its implementation mechanism is a centralized Worker, which guarantees lock consistency. Before using device resources, a work unit must acquire the lock. The lock state is consistent globally and can only be changed atomically, ensuring exclusive access to the resource. After acquiring the lock, if the resource is in an unloaded state, the work unit automatically loads the resource by calling the onload function; after the task is completed, the work unit releases the lock and unloads the resource through the offload function. The device lock uses the data dependency information of the data channel to define the acquisition priority, that is, a child work unit can only acquire the lock after the parent work unit enqueues data and releases the lock, thereby avoiding lock contention and deadlock. Furthermore, this mechanism, combined with the controller's device placement information, avoids unnecessary resource loading and unloading operations when working between different devices. Since the acquisition and release of this lock are accompanied by the state restoration or unloading of components, the acquisition or release time of the lock itself is generally negligible in comparison and will not cause a performance bottleneck. If the Worker corresponding to the lock fails, training can be resumed through the system checkpoint mechanism, which will be described later.

[0045] Macro-to-microfluidic conversion execution: In some embodiments, Figure 2 This demonstrates the execution management process of macro-to-microflow transformation. Workflows are written in an imperative manner, for example, using a for loop that iterates through the main logic to contain three work units: rollout, inference, and training. Rollout tasks, such as `generate(data, ch1)`, process data and enqueue the results into channel `ch1`. The execution flow manager can segment input data into smaller chunks based on the execution mode, resource allocation, and number of reinforcement learning components specified by the scheduler or user. This allows work units to process outputs with finer data granularity, or to merge tasks into larger-scale subtasks for differentiated time scheduling. By default, data is transmitted in the smallest unit capable of a complete data iteration, according to the proportion of components specified by the scheduler or user, ensuring data can move to the next stage as quickly as possible once ready. Simultaneously, device locks support automatic resource management between work units with data dependencies, achieving time-based scheduling through context switching. The execution mode is initially determined by the scheduler or user and is adjusted during operation through the "Adaptive Rescheduling for Runtime Dynamics" section.

[0046] Through macro-microflow transformation, user-defined workflows can be orchestrated within a complete temporal and spatial space. Figure 3 Representative execution patterns suitable for different reinforcement learning workloads and configurations are demonstrated: 1. Time-based scheduling mode: Each work unit exclusively uses all accelerators, and is swapped out after its task is completed, with the next work unit taking over. This mode is particularly effective when a work unit must occupy all the equipment (such as for training large models), but attention should be paid to the GPU idleness problem caused by the long tail effect.

[0047] 2. Space scheduling mode: Work units are allocated to independent GPUs, and pipelined pipelines reduce waiting time and achieve efficient resource alignment.

[0048] 3. Hybrid scheduling mode: For complex reinforcement learning workflows, some work units are distributed across multiple GPUs in a pipeline manner for execution. After a stage is completed, they are swapped out and replaced by subsequent work units to balance performance and resource efficiency.

[0049] Scheduling strategy: In some embodiments, to find the optimal execution mode within a large scheduling space, the system introduces analyzer and scheduler modules. The analyzer measures and estimates the execution characteristics of each component under different numbers of GPUs, including execution time and memory usage. The scheduler uses this data to assemble an overall execution plan. The analyzer uses polynomial extrapolation to estimate execution performance at larger scales and outputs an execution time estimation function. , as input to the scheduler.

[0050] The analyzer builds a theoretical model based on the computational characteristics of the load. First, it constructs an undetermined coefficient relationship between performance and computational load. Then, it uses actual runtime data to fit and determine the specific coefficients. Taking the Large Language Model (LLM) as an example, its theoretical model indicates that the prefill time for a single request has a quadratic relationship with the prefix sequence length, while the decoding time has a linear relationship with the prefix sequence length. That is: Where t is the prefix sequence length and C is the coefficient to be determined, which is generally obtained by regression from the actual data collected during the pre-measurement process. In addition, to address complex runtime statistical characteristics such as the long-tail effect, the analyzer will be dynamically calibrated and updated based on historical statistical data, thereby ensuring the accuracy of performance estimation in large-scale clusters.

[0051] The scheduling algorithm recursively divides the workflow graph into two interconnected subgraphs. and And evaluate the costs of time scheduling and space scheduling: In time scheduling, the cost is and The sum of costs plus resource switching overhead.

[0052] In space scheduling, the running time is estimated as follows: in, The larger of the preheating time and cooling time. The slowest subgraph running time, For the total batch, To handle granularity.

[0053] This algorithm is based on the ST-Cut algorithm in graph theory. It selects the optimal scheduling scheme and applies it recursively until the subgraph is reduced to a single node. The pseudocode of the algorithm is shown in Table 3 below: Table 3 Adaptive communication: In some embodiments, the system communication layer is designed to achieve: (1) flexibility: shielding the physical location and logical differences of components to enable communication between any components; (2) adaptability: adapting to heterogeneous data across nodes and devices and maximizing link throughput.

[0054] To address the limitations of traditional communication libraries (such as the NVIDIA Collective Communications Library, NCCL) in dynamic reinforcement learning scenarios, this system designs a communication protocol and primitives for perceiving work units and data placement. At the protocol level, the system features transparent connection lifecycle management, supporting dynamic placement and scaling. At the primitive level, the system automatically selects the optimal backend based on the work unit type of the communication device (e.g., NCCL for inter-GPU communication, CUDAIPC (CUDA Inter-Process Communication) within GPUs, and Gloo for inter-CPU communication), and supports structure-aware communication of any Python object. Furthermore, by constructing a high-level data channel (FIFO queue), the system achieves complete decoupling of control flow and data flow between producers and consumers, and supports custom scheduling based on load balancing strategies. It should be noted that the method of automatically selecting the optimal backend based on device type is the optimal solution; for example, NCCL is always superior to other solutions for inter-GPU communication. However, NCCL is not a general-purpose communication library; it must be used for Tensors on GPUs and cannot be used between communicators located on the same GPU. CUDAIPC can only be used between communicators on the same GPU. The "dynamic nature of reinforcement learning" here refers to the simultaneous occurrence of data requiring different communication methods during the learning process; the communication primitives here are designed to emphasize automation, that is, without the need to explicitly specify the communication backend.

[0055] Adaptive rescheduling for runtime dynamics: In some embodiments, reinforcement learning training is inherently dynamic. Taking a large language model as an example, as the model learns, the response length distribution of Rollout may drift, causing the execution speed of components such as text generation to increase or decrease over time. Therefore, a static execution plan derived from initial performance analysis may degenerate into a suboptimal solution during training.

[0056] To address this issue, the system embeds a runtime performance analyzer within each work unit to continuously record the execution time of each call; it also introduces a lightweight control logic responsible for comparing these actual measurements with the execution time estimated by the scheduler.

[0057] When the deviation between the two exceeds a configurable threshold, the system saves the checkpoint of the current training state, reruns the scheduling algorithm using the updated performance data, and resumes training under the new execution plan.

[0058] It should be noted that in typical inference reinforcement learning training, the drift in response length is gradual, and rescheduling is usually required only after thousands of iterations. Therefore, the overhead of saving checkpoints and redeploying is negligible relative to the total training time.

[0059] In some embodiments, the system incorporates both static resource constraint reservation and dynamic monitoring rollback mechanisms to ensure resource isolation and system stability.

[0060] First, during the static scheduling and planning phase, the aforementioned analyzer analyzes and records the GPU memory and system memory usage characteristics under different loads. When generating hybrid execution plans, the scheduler uses these memory measurements as hard constraints. The system pre-reserves GPU memory by calculating the theoretical peak memory usage of the concurrent pipeline and time-division multiplexing components, ensuring that the total GPU memory requirements of tasks scheduled on the same GPU are always met, thereby avoiding GPU memory shortage issues on the scheduling side.

[0061] Secondly, to address the risk of sudden memory shortages caused by dynamic increases in sequence length or exacerbated memory fragmentation due to context switching during runtime, the system reuses the checkpoint mechanism from "adaptive rescheduling" to implement real-time conflict detection and rollback strategies. Lightweight monitoring routines within the work unit track actual memory usage in real time; once a deviation from expected real-time memory usage is detected, the system immediately triggers a rollback mechanism. At this point, the system pauses current execution, discards abnormal steps, restores the training state from the most recently saved checkpoint, and feeds back the latest memory usage data to the analyzer to trigger rescheduling. Finally, training resumes in the updated execution mode.

[0062] The following is for reference Figure 4 , Figure 5 , Figure 6 The data compares the training throughput performance of this system with two current state-of-the-art open-source baseline systems, veRL and Slime, on a large language model reinforcement learning task. The data shows that the training throughput of this system is generally superior to the baseline systems, with improvements of up to 70%.

[0063] Figure 7 The study demonstrates a comparison of the system's training throughput performance under different execution modes in embodied intelligence reinforcement learning tasks. On ManiSkill and LIBERO, the system achieved throughput improvements of up to 87.2% and 143.4%, respectively.

[0064] Tables 4, 5, and 6 below show a comparison of the performance metrics of the models trained by this system under the above tasks.

[0065] Table 4 Table 5 Table 6 in Figure 4 The system demonstrates that the performance of its 1.5B and 7B-scale large language models trained on this system is on par with or exceeds that of the best baseline model for models of the same scale. Figure 5 and Figure 6 The final performance of the system trained on ManiSkill and LIBERO is shown respectively. The data shows that the model performance has increased significantly after training and exceeds the performance of the current state-of-the-art baseline model.

[0066] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0067] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0068] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0069] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0070] The background section of this invention may include background information about the problems or environment in which the invention is being developed, and is not necessarily a description of prior art. Therefore, the content included in the background section does not constitute an admission of prior art by the applicant.

[0071] The above description provides a further detailed explanation of the present invention in conjunction with specific / preferred embodiments, and it should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various substitutions or modifications can be made to these described embodiments without departing from the concept of the present invention, and all such substitutions or modifications should be considered within the scope of protection of the present invention. In the description of this specification, the reference to terms such as "an embodiment," "some embodiments," "preferred embodiment," "example," "specific example," or "some examples," etc., indicates that the specific features, structures, materials, or characteristics described in connection with that embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples. Without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification and the features of different embodiments or examples. Although the embodiments of the present invention and their advantages have been described in detail, it should be understood that various changes, substitutions, and modifications can be made herein without departing from the scope of protection of the patent application.

Claims

1. A resilient computing system for reinforcement learning, used to orchestrate and execute reinforcement learning workflows on heterogeneous computing device clusters, characterized in that, include: Multiple working units, each of which encapsulates a reinforcement learning component, the working unit supports resource offloading on demand and provides a communication primitive interface; A workflow executor is used to receive a macro logic flow defined by imperative programming that describes data interactions between multiple said work units; A scheduler, communicatively connected to the workflow executor, is used to convert the macro logic flow into a micro execution flow; An adaptive communication layer is used to perceive the physical placement information of multiple work units, manage the connection lifecycle between multiple work units, and automatically select a communication backend based on the physical placement information to support data transmission between multiple work units; wherein, the scheduler converts the macro logic flow into micro execution flow by: dynamically distributing the work units at a granular level in the spatial dimension through an elastic pipeline mechanism, and performing resource time-division multiplexing of the work units in the temporal dimension through a context switching mechanism.

2. The elastic computing system for reinforcement learning according to claim 1, characterized in that, The multiple work units inherit from the same base class, which provides send and receive primitives for communication between the multiple work units, as well as load and unload functions for resource management.

3. The elastic computing system for reinforcement learning according to claim 1, characterized in that, The scheduler also includes an analyzer module for establishing an execution time model for the work unit. The scheduler is configured to dynamically generate the optimal execution plan by recursively dividing the workflow graph based on the execution time model, thereby evaluating the cost of using spatial scheduling and time scheduling strategies for each subgraph generated after the workflow graph is divided. The analyzer module uses a polynomial extrapolation method to establish an execution time model for the work unit under different numbers of computing devices.

4. The elastic computing system for reinforcement learning according to claim 1, characterized in that, The elastic computing system for reinforcement learning also includes a data channel, which is a first-in-first-out queue used to decouple the control flow and data flow among multiple working units that have data dependencies on each other.

5. The elastic computing system for reinforcement learning according to claim 4, characterized in that, The time-division multiplexing of resources for the work unit through a context switching mechanism includes: restricting concurrent access to multiple work units with data flow dependencies on the same computing device in the heterogeneous computing device cluster through a distributed device lock; wherein, the work unit acquires the distributed device lock before accessing the resource; the acquisition priority of the distributed device lock is defined by the data dependency information between work units in the data channel.

6. The elastic computing system for reinforcement learning according to claim 1, characterized in that, The adaptive communication layer is configured to support multiple communication backends, including a collection communication library backend for inter-GPU communication, an inter-process direct memory access backend for intra-GPU communication, and a distributed communication framework backend for inter-CPU communication, and can automatically select the optimal communication backend based on the physical placement information.

7. The elastic computing system for reinforcement learning according to claim 1, characterized in that, The flexible pipeline mechanism dynamically adjusts the data processing granularity input to the work unit to achieve dynamic granularity distribution of the work unit; wherein, the work unit is configured to process data at different batch sizes, and when the output data of the current work unit reaches a configurable batch size, the batch data is forwarded to the next work unit, so that the next work unit can start early or delay processing at a granularity corresponding to the batch size.

8. The elastic computing system for reinforcement learning according to claim 1, characterized in that, The elastic computing system for reinforcement learning also includes a runtime monitor for continuously recording the actual execution characteristics of the work unit; when the deviation between the actual execution characteristics and the estimated execution characteristics on which the scheduler is based exceeds a threshold, the scheduler is triggered to reschedule to generate an updated micro-execution flow.

9. A reinforcement learning workflow orchestration method, running on a heterogeneous computing device cluster, characterized in that, The reinforcement learning workflow orchestration method includes: defining a macro logic flow describing data interaction between multiple work units through imperative programming, wherein each work unit encapsulates a reinforcement learning component and supports on-demand resource unloading; converting the macro logic flow into a micro execution flow; sensing the physical placement information of the multiple work units, and automatically selecting a communication backend to manage data transmission based on the physical placement information; wherein, the conversion of the macro logic flow into a micro execution flow includes: dynamically distributing the work units at a granular level in the spatial dimension through an elastic pipeline mechanism, and performing time-division multiplexing of resources for the work units in the temporal dimension through a context switching mechanism.

10. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, the reinforcement learning workflow orchestration method as described in claim 9 is implemented.

Images