Systems, methods, apparatuses, media, and program products for data processing

Abstract

A system, method, apparatus, medium, and program product for data processing is provided. The system includes: a first layer providing multiple operation interfaces to support various types of interactive operations, configured to receive a first operation instruction conforming to a first data format using a first operation interface among the multiple operation interfaces, the first operation instruction being used to request a first environment instance among multiple environment instances to perform a first type of operation; a second layer providing multiple adapters corresponding to the multiple environment instances, configured to convert the first operation instruction into a second operation instruction conforming to a second data format using the first adapters; and a third layer providing multiple environment interfaces corresponding to the multiple environment instances, configured to use the first environment interface to call the first environment instance based on the second operation instruction to perform the first type of operation, thereby obtaining a first execution result conforming to a second data format.

Description

Technical Field

[0001] The exemplary embodiments disclosed herein generally relate to the field of information technology, and particularly to methods, apparatus, devices, computer-readable storage media, and computer program products for resource scheduling. Background Technology

[0002] In reinforcement learning training systems, agents need to continuously interact with the environment to learn optimal policies. The environment is responsible for receiving the agent's output actions, executing state transitions, and returning observations and rewards. Depending on the task type, environments can include various types, such as simulation environments built based on physics simulation engines, execution environments for running tools, and verification environments for validating inference results. Different types of environments differ significantly in data formats, action spaces, and interaction modes. Existing solutions lack unified adaptation support for multiple environment types, requiring separate development of adaptation code for each type, thus hindering improvements in environment integration and training efficiency. Summary of the Invention

[0003] In a first aspect of this disclosure, a system for data processing is provided. The system includes: a first layer providing multiple operation interfaces for supporting various types of interactive operations, the first layer being configured to receive a first operation instruction conforming to a first data format using a first operation interface among the multiple operation interfaces, the first operation instruction being used to request a first environment instance among multiple environment instances to perform a first type of operation; a second layer providing multiple adapters corresponding to the multiple environment instances, the second layer being configured to convert the first operation instruction into a second operation instruction conforming to a second data format using a first adapter, the first adapter being an adapter among the multiple adapters corresponding to the first environment instance; and a third layer providing multiple environment interfaces corresponding to the multiple environment instances, the third layer being configured to use the first environment interface to call the first environment instance based on the second operation instruction to perform the first type of operation, thereby obtaining a first execution result conforming to a second data format.

[0004] In a second aspect of this disclosure, a method for data processing is provided. This method is executed by the system of the first aspect of this disclosure. The method includes: receiving a first operation instruction conforming to a first data format using a first operation interface among a plurality of operation interfaces, the first operation instruction being used to request a first environment instance among a plurality of environment instances to perform a first type of operation; converting the first operation instruction into a second operation instruction conforming to a second data format using a first adapter among a plurality of adapters, the first adapter corresponding to a first environment instance; and invoking the first environment instance to perform the first type of operation based on the second operation instruction using the first environment interface among a plurality of environment interfaces, to obtain a first execution result conforming to the second data format.

[0005] In a third aspect of this disclosure, an electronic device is provided. The electronic device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method of the second aspect of this disclosure when executed by the at least one processing unit.

[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. This computer-readable storage medium stores a computer program that can be executed by a processor to implement the method according to a second aspect of this disclosure.

[0007] In a fifth aspect of this disclosure, a computer program product is provided, which is tangibly stored in a computer storage medium and includes computer-executable instructions that, when executed by a device, cause the device to perform the method according to a second aspect of this disclosure.

[0008] In this way, through the unified operation interface of the first layer, the adapter conversion of the second layer, and the environment interface call of the third layer, the underlying differences of heterogeneous environments such as simulation environment, execution environment, and verification environment can be shielded, so that various types of environment instances can be accessed and called in a unified manner. The access of new environments does not require modification of the core code of the upper-layer training framework. At the same time, it provides an architectural foundation for the decoupling of environment execution and policy inference, the life cycle management of environment instances, and the combination and nesting of environments, thereby improving the efficiency and scalability of data processing.

[0009] It should be understood that the content described in this section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0010] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein: Figure 1 A schematic diagram of the example environment is shown; Figure 2 Block diagrams of systems for data processing in some scenarios are shown; Figure 3 The diagram illustrates the runtime state switching for some scenarios of an environment instance. Figure 4 Flowcharts illustrating example processes for data processing in several scenarios are shown; and Figure 5 A block diagram of an electronic device capable of implementing multiple illustrative scenarios is shown. Detailed Implementation

[0011] The examples in the text will now be described in more detail with reference to the accompanying drawings. While some examples are shown in the drawings, it should be understood that solutions can be implemented in various forms and should not be construed as limited to the examples presented herein. Rather, these examples are provided to provide a more thorough and complete understanding of the solutions. It should be understood that the drawings and examples in this document are for illustrative purposes only and are not intended to limit the scope of protection of the solutions.

[0012] It should be noted that the headings of any section / subsection provided herein are not restrictive. Various examples are described throughout this document, and examples of any type may be included under any section / subsection. Furthermore, examples described in any section / subsection may be combined in any way with any other examples described in the same section / subsection and / or different sections / subsections.

[0013] In the description of the examples in this document, the term "including" and similar terms should be understood as open inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "an example" or "the example" should be understood as "at least one example". The term "some examples" should be understood as "at least some examples". Other explicit and implicit definitions may also be included below. The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0014] The examples in this document may involve user data, data acquisition, and / or use. All of these aspects comply with relevant laws, regulations, and rules. In the examples, all data collection, acquisition, processing, manipulation, forwarding, and use are conducted with the user's knowledge and confirmation. Accordingly, when implementing each example, the type, scope of use, and usage scenarios of any data or information that may be involved should be communicated to the user and their authorization obtained through appropriate means, in accordance with relevant laws and regulations. The specific methods of notification and / or authorization can vary depending on the actual situation and application scenario; the scope of the solution is not limited in this regard.

[0015] In this manual and the sample solutions, any processing of personal information will be conducted only under legal grounds (such as obtaining the consent of the data subject or being necessary for the performance of a contract) and will only be carried out within the scope stipulated or agreed upon. A user's refusal to process personal information beyond what is necessary for basic functions will not affect the user's use of basic functions.

[0016] As used in this paper, the term "model" refers to a model that learns the relationship between inputs and outputs from training data, enabling it to generate corresponding outputs for a given input after training. Model generation can be based on machine learning techniques. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs using multiple layers of processing units. A neural network model is an example of a deep learning-based model. In this paper, "model" may also be referred to as a "machine learning model," "learning model," "machine learning network," or "learning network," and these terms are used interchangeably.

[0017] Machine learning typically comprises three phases: training, testing, and application (also known as inference). In the training phase, a given model is trained using a large amount of training data, iteratively updating its parameter values ​​until the model can consistently generate inferences that meet the expected goals from the training data. Through training, the model can be considered to have learned the relationship between inputs and outputs (also known as the input-output mapping) from the training data. The parameter values ​​of the trained model are determined. In the testing phase, test inputs are applied to the trained model to test whether it can provide the correct output, thus determining the model's performance. In the application phase, the model can be used to process actual inputs based on the trained parameter values ​​to determine the corresponding output.

[0018] As mentioned above, in reinforcement learning training systems, agents need to continuously interact with the environment to learn optimal policies. The environment is responsible for receiving the actions output by the agent, executing state transitions, and returning observations and rewards. Depending on the task type, reinforcement learning environments can be categorized into several types. For example, simulation environments are built based on physics simulation engines and are used for tasks such as robot manipulation, involving multimodal observation data and continuous action spaces. Execution environments are built based on sandboxes, browser-based automation tools, etc., where agents complete complex tasks by interacting with external tools. Verification environments are built based on symbolic computation engines or formal provers, receiving inference steps as actions and using a verification engine to determine the correctness of the inference.

[0019] The aforementioned different types of environments differ significantly in terms of observation data format, action space, and interaction modes. Simulation environments observe multimodal tensors, execution environments observe document object model trees or structured data, and validation environments observe proof states. Traditional training frameworks lack a unified interface abstraction layer to shield these underlying differences, requiring the writing of adaptation code from scratch for each new environment. Furthermore, the lack of an effective decoupling mechanism between environment execution and policy inference limits the utilization of computational resources. The creation, warm-up, reuse, and recycling of environment instances also lack standardized lifecycle management mechanisms, resulting in long cold start times for environment instances.

[0020] In view of this, this paper proposes a scheme for data processing. According to this scheme, the data processing system adopts a layered architecture design, including a first layer, a second layer, and a third layer. The first layer provides multiple operation interfaces to support various types of interactive operations. Using a first operation interface among these interfaces, a first operation instruction conforming to a first data format is received. This first operation instruction is used to request a first environment instance among multiple environment instances to execute a first type of operation. The second layer provides multiple adapters corresponding to the multiple environment instances. Using the first adapter corresponding to the first environment instance, the first operation instruction is converted into a second operation instruction conforming to a second data format. The third layer provides multiple environment interfaces corresponding to the multiple environment instances. Using the first environment interface, the first environment instance is invoked based on the second operation instruction to execute the first type of operation, thereby obtaining a first execution result conforming to the second data format.

[0021] In this way, the underlying differences between heterogeneous environments are shielded by a unified operation interface, the conversion between a unified data format and an environment-specific data format is achieved through an adapter, and the underlying environment instance is connected through the environment interface. This enables various types of environment instances, such as physical simulation, sandbox tools, and inference verification, to be accessed and called in a unified manner. New environment access does not require modification of the core code of the training framework, thus improving the efficiency, scalability, and maintainability of data processing.

[0022] The following describes various examples of this scheme in further detail with reference to the accompanying drawings.

[0023] Figure 1 A schematic diagram of example environment 100 is shown. (e.g.) Figure 1 As shown, example environment 100 may include, for example, a system for performing reinforcement learning training tasks (e.g., it may be referred to as a training system). In some examples, the training system may adopt a layered and decoupled architecture, which may include, from top to bottom, an access layer 110, a scheduling layer 120, an adaptation layer 130, an engine layer 140, a data layer 150, and a computing resource pool 160.

[0024] Access layer 110 is a unified task access and representation layer, configured to receive and represent reinforcement learning tasks, and is located at the top layer of the architecture. In some examples, access layer 110 can receive task requests for reinforcement learning tasks and parse the task requests into standardized task descriptions. The task descriptions may include, but are not limited to, at least one of the following: task type identifier, model configuration, environment configuration, resource requirement description, and recommended hyperparameters.

[0025] The scheduling layer 120 is configured to orchestrate and schedule reinforcement learning tasks. In some examples, the scheduling layer 120 can determine the task execution plan (e.g., a directed acyclic graph) of the reinforcement learning task based on the task description provided by the access layer 110. The scheduling layer 120 can also be used to perform full lifecycle management of reinforcement learning tasks. In some examples, the scheduling layer 120 can determine the task type and at least one task stage of the reinforcement learning task based on the task execution plan, thereby determining a scheduling scheme indicating computing resources, and scheduling computing resources in the computing resource pool 160 based on the scheduling scheme.

[0026] The adaptation layer 130 is configured to provide a unified environment access interface for different types of reinforcement learning tasks, adapting to the underlying differences between different environments such as physical simulation environments, tool calling environments, and logical reasoning verification environments. The engine layer 140 is configured to provide the computational capabilities required for reinforcement learning tasks, such as training, inference, and environment execution. The data layer 150 is configured to provide data services required for reinforcement learning tasks, such as sample and trajectory management, reward calculation and aggregation, and model parameter synchronization.

[0027] Computational resource pool 160 is configured to provide computing resources. In some examples, such as... Figure 1 As shown, the computing resource pool 160 may include a first resource cluster, a second resource cluster, a third resource cluster, and a fourth resource cluster. The first resource cluster may, for example, include a training cluster composed of graphics processing units (GPUs); the second resource cluster may, for example, include an environment cluster composed of GPUs; the third resource cluster may, for example, include an inference cluster composed of GPUs; and the fourth resource cluster may, for example, include a sandbox cluster composed of central processing units (CPUs). It should be understood that the types and numbers of the resource clusters described above are merely exemplary and can be flexibly configured according to actual needs.

[0028] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of the scheme. The example will continue to be described below with reference to the accompanying drawings.

[0029] The following description of the example will continue with reference to the accompanying drawings.

[0030] Figure 2 A block diagram of a system 200 for data processing is shown, depending on several scenarios. System 200 can be implemented in the example environment 100, and in some cases, system 200 can be implemented as a training system or as part of a training system, such as adaptation layer 130. For ease of description, the following text will use training system 200 as an example.

[0031] like Figure 2As shown, the training system 200 may include a first layer 210, a second layer 220, an asynchronous interaction protocol layer 230, and a third layer 240. The first layer 210 provides multiple operation interfaces to support various types of interactive operations. The second layer 220 provides multiple adapters corresponding to multiple environment instances. The third layer 240 provides multiple environment interfaces corresponding to multiple environment instances. The asynchronous interaction protocol layer 230 provides queues and controllers to support asynchronous interaction between policy inference and environment execution. These are described in detail layer by layer below.

[0032] In some cases, the first layer 210 is configured to receive a first operation instruction conforming to a first data format using a first operation interface among multiple operation interfaces. This first operation instruction is used to request a first environment instance among multiple environment instances to perform a first type of operation. For example, the first layer 210 can be a unified environment interaction interface layer (also called a unified interface layer or EIA layer), designed to shield the differences between the underlying heterogeneous environments and provide a unified environment invocation method for the upper-layer training framework. The term "first data format" used in this paper refers to the unified data format provided externally by the first layer 210, which is independent of the specific type of the underlying environment. The term "second data format" refers to the environment-specific data format corresponding to the corresponding environment instance, such as the data format natively supported by the underlying environment engine.

[0033] like Figure 2 As shown, the multiple operation interfaces may include at least one of the following: reset interface 211, step interface 212, rendering interface 213, reward determination interface 214, termination determination interface 215, and shutdown interface 216.

[0034] The reset interface 211 is configured to receive reset instructions, such as the "reset()" interface. The reset instruction requests that at least one environment instance out of a plurality of environment instances be restored to its initial state, and returns the first observation data of that at least one environment instance in its initial state. For example, by calling the reset interface 211, the environment can be reset to its initial state and the initial observations can be returned.

[0035] Step interface 212 is configured to receive step instructions, such as the "step()" interface. The step instruction carries first action information, instructing at least one environment instance to perform a first action based on the first action information to update the environment state of that at least one environment instance, and to provide feedback related to the first action. For example, by calling step interface 212, an action can be sent to the environment, performing a one-step state transition and returning a feedback result.

[0036] Rendering interface 213 is configured to receive rendering instructions, such as the "render()" interface. These instructions specify the rendering mode and request visualization information for at least one environment instance. The visualization information indicates the updated environment state of the at least one environment instance under the specified rendering mode. Rendering interface 213 can support multiple rendering modes, such as RGB array, depth map, and human visualization rendering modes.

[0037] The reward determination interface 214 is configured to receive reward determination instructions, such as the "compute_reward()" interface, which instructs the determination of the first reward information corresponding to the updated environment state. The reward determination interface 214 can calculate the reward value based on the current state and can support customizable reward functions.

[0038] The termination determination interface 215 is configured to receive a termination determination instruction, such as the "is_terminated()" interface, which instructs the determination of whether the task related to the first action is in a terminated state, in order to obtain a determination result. For example, the termination determination interface 215 can determine whether the current round terminated due to success or failure.

[0039] The close interface 216 is configured to receive a close instruction, such as the “close()” interface, which instructs to close at least one environment instance and release the computing resources associated with the at least one environment instance.

[0040] In some cases, the feedback result includes at least one of the following: a first field (e.g., the `obs` field), indicating second observation data for at least one environment instance, indicating the updated environment state of that at least one environment instance. Depending on the environment type, the second observation data can be a tensor, a DOM tree, structured data, or proof state, and its data type can be defined by the ObservationSpec; a second field (e.g., the `reward` field), indicating second reward information for the first action, such as the reward value for the current step, and its data type can be floating-point; a third field (e.g., the `terminated` field), indicating whether the task related to the first action is in a terminated state, such as indicating whether the round is terminated due to success or failure, and its data type can be boolean; a fourth field (e.g., the `truncated` field), indicating whether the task related to the first action is in a truncated state, such as indicating whether the round is truncated due to external conditions such as step limits, and its data type can be boolean; and a fifth field (e.g., the `info` field), indicating auxiliary information related to at least one environment instance, such as an auxiliary information dictionary containing environment-specific debugging information. All adapters can return the above unified five-field format after calling the step operation, ensuring that the upper-layer training framework does not need to care about the differences in the underlying environment type.

[0041] In some scenarios, the first layer 210 can provide observation formats for observation data and action formats for action information, defining data formats for different types of environments. For example, for a simulation environment, observation data may include RGB image tensors, depth maps, point cloud data, joint angles, torque data, etc., while action information may include continuous vectors (e.g., motor torque, joint position targets, etc.). For an execution environment, observation formats may include DOM trees, screenshots, text content, API responses, etc., while action information may include structured data (e.g., click coordinates, input text, API call parameters, etc.). For a verification environment, observation data may include the current proof state, the target to be proven, etc., while action information may be a text sequence (e.g., reasoning steps, etc.).

[0042] In some cases, multiple environment instances include at least one of a first type, a second type, and a third type. The first type of environment instance is configured to provide a simulation environment for simulating a physical environment, such as an environment instance built based on a physics simulation engine. The second type of environment instance is configured to provide an execution environment for running tools, such as an environment instance built based on a container sandbox, browser automation, network interface calls, or tool interaction protocols. The third type of environment instance is configured to provide a verification environment for verifying the correctness of the inference results of a machine learning model, such as an environment instance built based on a symbolic computation engine or a formal prover.

[0043] In some cases, the unified interface of Layer 1 210 can also be implemented in other ways. For example, a Remote Procedure Call (RPC) framework and cross-language serialization protocol can be used to define a cross-language environment interface standard, replacing native method calls in programming languages ​​to support environment engines written in heterogeneous languages. Another example is the use of a general reinforcement learning environment base class as the foundation for the unified interface, upon which multimodal observation and asynchronous interaction support can be extended. This paper does not impose any restrictions on this approach.

[0044] Next, the second layer 220 is described. The second layer 220 provides multiple adapters corresponding to multiple environment instances. The second layer 220 is configured to use a first adapter to convert a first operation instruction into a second operation instruction conforming to a second data format, the first adapter being the adapter corresponding to the first environment instance among the multiple adapters. For example... Figure 2 As shown, in some cases, the second layer 220 includes an abstraction layer 221 and a simulation environment adapter 222, an execution environment adapter 223, a verification environment adapter 224, and a composition adapter 225 that inherit from the abstraction layer 221.

[0045] Abstraction layer 221 defines the common methods that all adapters must implement, such as abstract reset methods, abstract stepping methods, abstract rendering methods, abstract shutdown methods, as well as data transformation methods, reward normalization methods, and configuration validation methods. Each dedicated adapter inherits from abstraction layer 221 and encapsulates the corresponding type of environment engine.

[0046] The simulation environment adapter 222 (also known as the embodied environment adapter) encapsulates one or more physical simulation engines. In some cases, the simulation environment adapter 222 uses Unified Memory technology to achieve zero-copy data transfer between the Graphics Processing Unit (GPU) and the physical simulation engine, avoiding the overhead of copying observation data back and forth between the Central Processing Unit (CPU) and the GPU. Unified Memory is a memory technology that allows the CPU and GPU to share the same virtual address space. In some cases, the simulation environment adapter 222 supports unified encoding of multimodal observation data such as RGB images, point clouds, joint angles, and torques. In some cases, the simulation environment adapter 222 supports GPU parallel batch stepping, processing stepping requests for multiple environment instances at once. In some cases, the simulation environment adapter 222 also provides a Domain Randomization configurator, supporting dynamic adjustment of physical parameters (such as friction coefficient, mass, and illumination) during training.

[0047] The Execution Environment Adapter 223 (also known as the Agent Environment Adapter) encapsulates container sandboxes, browser automation, network API calls, and tool interaction protocols. In some cases, the container sandbox supports secure code execution environment isolation, with each environment instance running in an independent container. In some cases, browser automation supports operations such as DOM tree parsing, element location, and click input, with observations presented as DOM snapshots and screenshots. In some cases, network API tool calls support network requests, with actions encoded as structured data in the form of API call parameters. In some cases, tool interaction protocol adaptations support tool interactions compatible with large model context protocols. In some cases, the Execution Environment Adapter 223 also provides a pre-warmed container pool mechanism, which creates and pre-warms containers in advance, allowing the environment state to recover within, for example, 2 seconds, significantly reducing cold start latency.

[0048] The verification environment adapter 224 (also known as the inference verification adapter) encapsulates the symbolic computation engine and the formal prover. In some cases, the symbolic computation engine is responsible for verifying symbolic mathematical computations (e.g., expression simplification, equation solution correctness checks). In other cases, the formal prover is responsible for verifying formal mathematical proofs, receiving inference steps as actions and returning proof status updates. In some cases, the verification environment adapter 224 also supports executing verification of code generation tasks using the compiler. In some cases, the verification environment adapter 224 also provides a verifier instance cache mechanism, i.e., pre-creating a verifier process pool to avoid restarting the verifier on each call, reducing cold start time.

[0049] In some cases, multiple adapters may also include a combination adapter 225 (also referred to as a combination environment adapter). The combination adapter 225 is configured to perform tasks using at least two of the multiple environment instances, thereby supporting the combination and nesting of environments.

[0050] In some scenarios, the combination adapter 225 supports a first mode (also known as a parallel mode). In the first mode, the combination adapter 225 executes actions in parallel across at least two environment instances to update the environmental states of those instances, acquire at least two fifth observations corresponding to those instances, and aggregate these fifth observations to obtain a sixth observation. For example, multiple sub-environments execute independently and simultaneously, each receiving the same action or its own distinct actions. The observations from each sub-environment are merged into a joint observation (i.e., the sixth observation) via an observation aggregator, and the rewards from each sub-environment are merged into a joint reward (e.g., summation, weighted average, etc.) via a reward aggregator. The parallel mode is suitable for scenarios involving multi-robot collaboration and simultaneous use of multiple tools.

[0051] In some scenarios, the combination adapter 225 supports a second mode (also known as a serial mode). In this second mode, the combination adapter 225 executes actions sequentially in each of the at least two environment instances based on at least two action information corresponding to those instances, according to a predetermined order. The observation information of the preceding environment instance in the predetermined order is used to determine the action information of the following environment instance. For example, multiple sub-environments are executed sequentially in a predefined order, with the output (observation or processing result) of the preceding sub-environment serving as the input (action or context) of the following sub-environment. As an example, information can be searched first by the browser environment, and then the search results can be passed to the code execution environment to write the program. The serial mode is suitable for multi-step toolchain tasks.

[0052] In some cases, environment composition can also be achieved in other ways. For example, a multi-agent environment framework can be used as the basis for composition, leveraging its parallel agent interfaces to achieve environment composition. Alternatively, the composition logic can be moved up to a higher-level training coordinator (e.g., the orchestration layer of downstream modules), where the coordinator manages the interaction order and data aggregation of multiple independent environments, rather than implementing the composition adapter at the adaptation layer. This paper does not impose any restrictions on this approach.

[0053] In some cases, alternative implementations can be used for dedicated adapters. For example, for simulation environments, a unified physical simulation middleware interface can be used instead of directly connecting to each simulation engine. For execution environments, container groups from a container orchestration platform can be used as sandbox runtimes; or serverless functions can be used to execute lightweight tool calls. For verification environments, a remote verification service (Verification-as-a-Service) can be used instead of a local verifier process pool, calling external verification engines via network interfaces. This document does not impose any restrictions on this.

[0054] The asynchronous interaction protocol layer 230 is described next. Asynchronous interaction protocol layer 230 (also known as the multi-type environment asynchronous interaction protocol layer) aims to achieve pipelined parallelism between GPU-side policy inference and CPU-side environment execution. Asynchronous interaction protocol layer 230 adopts a producer-consumer queue model. For example... Figure 2 As shown, in some cases, the asynchronous interaction protocol layer 230 includes an action queue 231, an observation queue 232, a reward queue 233, a first controller 234, a second controller 235, and a third controller 236.

[0055] Accordingly, in some cases, the training system 200 also includes an action queue 231, an observation queue 232, an inference engine, and an environment execution engine. The action queue 231 stores second action information, which instructs at least one environment instance among multiple environment instances to execute a second action. The data direction of the action queue 231 is from GPU to CPU, used to cache actions inferred by the policy network. For example, the GPU-side inference thread writes actions in batches to the action queue 231, and the CPU-side environment execution thread retrieves actions from the action queue 231 and sends them to the environment instances. The observation queue 232 stores third observation information, which indicates the environment state of at least one environment instance. The data direction of the observation queue 232 is from CPU to GPU, used to cache observation data after environment execution. For example, the CPU-side environment execution thread writes completed observations to the observation queue 232, and the GPU-side inference thread retrieves observations from the observation queue 232 for the next round of inference.

[0056] The inference engine is configured to retrieve third observation information from observation queue 232, use a machine learning model to obtain third action information based on the third observation information, and add the third action information to action queue 231. The third action information indicates that a third action should be executed at least one environment instance. For example, the inference engine could be a policy network running on a GPU, which performs forward inference based on the current observations and outputs an action. The environment execution engine is configured to retrieve second action information from action queue 231, execute the second action at least one environment instance to update the environment state of the at least one environment instance, obtain fourth observation information, and add the fourth observation information to observation queue 232. The fourth observation information indicates the updated environment state of the at least one environment instance. For example, the environment execution engine could run on a CPU and be responsible for the actual execution of the environment logic.

[0057] In some cases, the environment execution engine is also configured to determine third reward information corresponding to the second action, and the training system 200 also includes a reward queue 233, which is configured to store the third reward information. The data direction of the reward queue 233 is from the CPU to the downstream reward calculation and feedback module, and it is used to cache the reward values ​​returned by the environment for reward statistics and policy gradient calculation.

[0058] In some cases, the asynchronous interaction protocol layer 230 also includes a step event bus. The step event bus is a publish-subscribe event bus used to coordinate the step timing of multiple environment instances, ensuring the consistency of the state of each environment instance during batch training.

[0059] In some cases, the training system 200 also includes a first controller 234. The first controller 234 (also referred to as a batch aggregation controller) is configured to aggregate third action information of a first number of items and add it to an action queue 231, or aggregate fourth observation information of a second number of items and add it to an observation queue 232. The first controller 234 aggregates actions and / or observations of multiple environment instances into batches for processing, thereby reducing GPU kernel function startup overhead and the communication frequency between the CPU and GPU. As an example, when actions for 32 parallel environments are ready, the first controller 234 can send these actions in batches to the environment execution engine for execution.

[0060] In some cases, the training system 200 also includes a second controller 235. The second controller 235 (also referred to as a timeout controller) is configured to determine a timeout threshold corresponding to the at least one environment instance based on its environment type, and to perform timeout processing in response to the at least one environment instance taking longer to execute a second action than the timeout threshold. Different environment types correspond to different timeout thresholds. The second controller 235 sets differentiated timeout thresholds for different environment types to prevent environment freezes that block the entire training pipeline. As an example, for a simulation environment, the timeout threshold can be set to, for example, 1000 milliseconds (physical simulation steps are typically fast). For an execution environment, the timeout threshold can be set to, for example, 10000 milliseconds (sandbox execution and browser operations may be slower). For a verification environment, the timeout threshold can be set to, for example, 500 milliseconds (symbolic computation and proof verification are typically fast). Timeout processing may include, for example, marking the timed-out environment instance as an anomalous state.

[0061] In some cases, the training system 200 also includes a third controller 236. The third controller 236 (also referred to as a backpressure controller) is configured to determine the fill rate of a target queue and, based on the fill rate, control the rate at which data is added to the target queue in the data supply direction corresponding to the target queue, which includes at least one of an action queue 231 and an observation queue 232. The term "backpressure" as used herein refers to a flow control mechanism that feeds a deceleration signal upstream when downstream processing capacity is insufficient. In some cases, the third controller 236 reduces the rate at which data is added to the target queue in the data supply direction in response to the fill rate exceeding a first threshold; and suspends the addition of data to the target queue in the data supply direction in response to the fill rate exceeding a second threshold, where the second threshold is greater than the first threshold. As an example, when the queue fill rate exceeds 80% (an example of the first threshold), the third controller 236 triggers flow control, reducing the upstream data generation rate. When the queue fill rate exceeds 95% (an example of the second threshold), the third controller 236 triggers a pause, temporarily stopping upstream writes to prevent queue overflow and memory explosion.

[0062] In some scenarios, the asynchronous interaction protocol layer 230 achieves pipelined parallelism between the GPU and CPU through the following six stages. During the GPU inference stage, the policy network on the GPU (i.e., the machine learning model in the inference engine) performs forward inference based on the current observations and outputs an action. This step is executed entirely on the GPU and does not involve interaction with the CPU environment.

[0063] During the action enqueueing phase, the inferred actions undergo post-processing (such as de-standardization and format conversion) before being written in batches to the action queue 231 by the GPU-side write thread. The first controller 234 is responsible for aggregating actions from multiple environments into a single batch, reducing communication overhead.

[0064] During the CPU environment stepping phase, the CPU-side environment execution thread retrieves actions in batches from action queue 231 and distributes them to the corresponding environment instances for stepping operations. During this phase, the GPU can simultaneously perform other training tasks (such as Critic network updates and gradient calculations).

[0065] During the observation data enqueueing phase, after the environment execution is complete, the CPU writes the returned observation data to the observation queue 232 and the reward value to the reward queue 233. The second controller 235 checks the execution time of each step, and marks the environment instance as abnormal if a timeout occurs.

[0066] During the GPU observation data retrieval phase, the GPU-side training main loop retrieves observation data in batches from observation queue 232, preparing for the next round of policy inference. The third controller 236 checks the queue fill rate and implements backpressure flow control.

[0067] During the iterative loop phase, the aforementioned five phases are repeated, forming a pipeline loop of "inference, enqueue, step, dequeue, and re-inference". Ideally, GPU inference time overlaps with CPU environment execution time, and the GPU can start the next batch of inference without waiting for the environment stepping to complete.

[0068] Through the above process, the asynchronous interaction protocol layer 230 realizes pipeline parallelism between the GPU and the CPU, eliminates GPU idle waiting in synchronous mode, and increases the overall GPU utilization rate from less than 40% to more than 75%.

[0069] In some situations, asynchronous interaction can also be implemented in other ways. For example, shared memory can be used instead of queues for data transfer between the GPU and CPU, further reducing communication latency. Another example is the use of an actor model from a distributed computing framework to replace a self-developed queue system, leveraging the framework's built-in scheduling to achieve asynchronous interaction. Yet another example is migrating all environment execution to the GPU (e.g., using a GPU-accelerated physics simulator) to achieve a full GPU pipeline, eliminating data transfer between the CPU and GPU. This paper does not impose any limitations on this approach.

[0070] Next, the third layer 240 is described. The third layer 240 provides multiple environment interfaces corresponding to multiple environment instances. The third layer 240 is configured to utilize the first environment interface to invoke the first environment instance based on a second operation instruction to execute a first type of operation, thereby obtaining a first execution result conforming to a second data format. For example... Figure 2 As shown, in some cases, the third layer 240 includes a simulation environment interface 241, an execution environment interface 242, a verification environment interface 243, a first manager 244, and a second manager 245.

[0071] In some cases, multiple environment interfaces encapsulate the low-level operation interfaces of various environment engines. For example, simulation environment interface 241 interfaces with the low-level interface of the physics simulation engine. Execution environment interface 242 (also known as the sandbox manager interface) interfaces with the container daemon and container runtime. Verification environment interface 243 (also known as the verification engine manager interface) interfaces with the verifier process pool.

[0072] In some cases, the training system 200 further includes an environment execution engine configured to manage multiple environment instances, and the third layer 240 further includes at least one of the following: a first engine interface configured to create an environment instance corresponding to the configuration information using the environment execution engine based on configuration information; a second engine interface configured to reset the environment instance corresponding to the first identification information using the environment execution engine based on a first identification information indicating the environment instance; a third engine interface configured to perform actions in a group of environment instances corresponding to the set of identification information based on a set of identification information and a set of action information using the environment execution engine to update the environment state of the group of environment instances; a fourth engine interface configured to obtain a set of observation information from the group of environment instances; and a fifth engine interface configured to close the environment instance corresponding to the second identification information based on a second identification information indicating the environment instance and release the computing resources occupied by the environment instance.

[0073] In some cases, the training system 200 also includes a first manager 244 and a second manager 245. The first manager 244 (also referred to as a warm pool manager) is configured to maintain a warm pool, which includes at least one pre-created environment instance. As used herein, the term "warm pool" refers to a pool of pre-created and maintained environment instances used to eliminate cold start latency. The second manager 245 (also referred to as a lifecycle manager) is configured to manage this at least one environment instance. The second manager 245 works in conjunction with the first manager 244 to manage the lifecycle of each environment instance according to a state machine.

[0074] In some cases, the second manager 245 is further configured to manage the at least one environment instance based on its runtime state, which includes at least one of the following: Warm-up state (WARM), indicating that the corresponding environment instance has completed initialization and is in an allocation-ready state, i.e., the environment instance has been warmed up and is ready to be allocated; Active state (ACTIVE), indicating that the corresponding environment instance has been allocated to the first task, i.e., the environment instance has been assigned to the training task and is participating in the interaction; Stable state (STALE), indicating that the corresponding environment instance is in an abnormal state, such as an execution exception (e.g., timeout, crash), and needs to be reclaimed; Recycling state (RECYCLING), indicating that the computing resources associated with the corresponding environment instance are in the process of cleanup, i.e., the environment instance is undergoing resource cleanup and is preparing to return to the warm-up state after rewarming; Destroyed state (DESTROYED), indicating that the computing resources associated with the corresponding environment instance have been released, i.e., the environment instance has been completely released and the resources have been fully reclaimed.

[0075] Figure 3 A schematic diagram 300 illustrates several scenarios for the transitions between runtime states of an environment instance. Diagram 300 shows the multiple runtime states an environment instance may experience during its lifecycle and the transition paths between these states. See below for reference. Figure 3 This describes the process of switching the runtime state of an environment instance.

[0076] In some scenarios, the second manager 245 is further configured to manage at least one environment instance based on its runtime state. The runtime state may include at least one of the following: a warm-up state 310, indicating that the corresponding environment instance has completed initialization and is in an allocation-pending state; an active state 320, indicating that the corresponding environment instance has been allocated to a first task; a failed state 330, indicating that the corresponding environment instance is in an abnormal state; a reclaim state 340, indicating that the computing resources associated with the corresponding environment instance are in the process of being cleaned up; and a destruction state 350, indicating that the computing resources associated with the corresponding environment instance have been released.

[0077] like Figure 3 As shown, the process of switching the runtime state of an environment instance is as follows. Upon system startup, the environment instance enters preheating state 310 after preheating. The environment instance in preheating state 310 has completed initialization, is located in the environment preheating pool, and awaits allocation to a training task.

[0078] When a training task requests an environment instance, the second manager 245 allocates an environment instance from the environment preheating pool, and the running state of the environment instance changes from preheating state 310 to active state 320. When the training task releases the environment instance, the second manager 245 reclaims the environment instance to the environment preheating pool, and the running state changes from active state 320 back to preheating state 310. This environment instance can then be reused by other tasks.

[0079] When an environment instance in active state 320 experiences an execution exception (such as step timeout or environment crash), its running state changes from active state 320 to inactive state 330, marking that the environment instance needs to be reclaimed. The second manager 245 initiates the reclamation process, transferring the environment instance in inactive state 330 to reclamation state 340, and begins resource cleanup.

[0080] When an environment instance in recycling state 340 completes resource cleanup, its running state transitions from recycling state 340 back to preheating state 310, and the environment instance re-enters the environment preheating pool to await reuse. When resource cleanup fails or times out, the running state transitions from recycling state 340 to destruction state 350, completely releasing resources.

[0081] Furthermore, environment instances in active state 320 can be forcibly terminated, and their running state directly transitions to destroyed state 350. Environment instances in warm-up state 310 can also automatically transition to destroyed state 350 when their idle time exceeds a threshold, releasing resources to avoid unnecessary resource occupation. Destroyed state 350 is the final state, triggering the final release of underlying resources.

[0082] Through the environment preheating pool and the aforementioned operational state switching mechanism, the cold start time of environment instances is reduced from a long time to a short time allocated from the preheating pool, achieving efficient reuse of environment instances in the training pipeline. Since environment instances can quickly switch between preheating state 310 and active state 320, the overhead of creating an environment instance from scratch for each use is avoided, reducing the impact of environment instance initialization on the training pipeline throughput. The introduction of failure state 330 and recycling state 340 enables the system to automatically detect and recover abnormal environment instances, reducing training interruption time caused by environment anomalies.

[0083] In some scenarios, lifecycle management of environment instances can be implemented in other ways. For example, a process-level fork mechanism can be used instead of an environment preheating pool, quickly creating new environment instances by copying an initialized parent process. Another example is the container checkpointing / recovery mechanism, which replaces a preheating container pool, saving the container state to disk and restoring it when needed. Yet another example is a simplified three-state machine that merges the preheating and active states, and the invalidation and recycling states, simplifying implementation but sacrificing some flexibility. This paper does not impose any restrictions on this approach.

[0084] In some scenarios, the training system 200 may also include an elastic scheduling layer 250. The elastic scheduling layer 250 is responsible for resource management, health checks, and elastic scaling of environment instances, serving as the interface layer between the training system 200 and the underlying infrastructure. In some scenarios, the elastic scheduling layer 250 includes at least one of a resource feature component, a health reporting component, and a scaling interface. The resource feature component maintains resource requirement characteristics for each environment instance, recording its typical CPU / GPU / memory usage, peak usage, and execution latency distribution. These resource characteristics guide scheduling decisions, such as scheduling environment instances with high resource requirements to high-configuration nodes. The health reporting component periodically (e.g., every 30 seconds) reports the health status of environment instances, including metrics such as liveness, recent step latency, and number of anomalies. When an environment instance anomaly is detected, the second manager 245 is triggered to perform a state transition (e.g., from active state 320 to failed state 330 and then to reclaimed state 340). The scaling interface provides a dynamic scaling mechanism for the number of environment instances. It automatically adjusts the number of instances in the environment warm-up pool based on the training load, automatically scaling up when training parallelism increases and automatically scaling down when parallelism decreases, optimizing resource utilization efficiency. In some scenarios, the state flow of the elastic scheduling layer 250 includes: discovery, registration, health check, scheduling allocation, inspection, and fault recovery / elastic scaling.

[0085] In some cases, the elastic scheduling layer 250 can also be implemented in other ways. For example, the horizontal auto-scaling component of a container orchestration platform can be used instead of a self-developed scaling interface, leveraging the native auto-scaling capabilities of the container orchestration platform. Another example is the use of an open-source inspection system instead of a self-developed health reporting component, utilizing mature open-source inspection solutions for environmental health monitoring. This article does not impose any restrictions on this approach.

[0086] After describing the layer structure of the training system 200, the data processing flow in the training system 200 is described below. The first layer 210 uses a first operation interface among multiple operation interfaces to receive a first operation instruction conforming to a first data format. The first operation instruction is used to request a first environment instance among multiple environment instances to execute a first type of operation. For example, the first operation instruction can be a step instruction received through step interface 212, and the first type of operation can be a step operation; or, for example, the first operation instruction can be a reset instruction received through reset interface 211, and the first type of operation can be a reset operation. The second layer 220 uses a first adapter corresponding to the first environment instance to convert the first operation instruction into a second operation instruction conforming to a second data format. For example, when the first environment instance is a simulation environment instance, the first adapter is simulation environment adapter 222, which converts operation instructions in a unified data format into operation instructions in a data format natively supported by the physics simulation engine. The third layer 240 uses the first environment interface to call the first environment instance based on the second operation instruction to execute the first type of operation, thereby obtaining a first execution result conforming to the second data format.

[0087] In some cases, the third layer 240 is further configured to provide a first execution result to the first adapter using the first environment interface; the second layer 220 is further configured to convert the first execution result into a second execution result conforming to the first data format using the first adapter; and the first layer 210 is further configured to feed back the second execution result in response to the first operation instruction using the first operation interface. In this way, the execution result is converted back to a unified data format via the adapter in the opposite direction and fed back to the upper-layer caller, thereby realizing complete bidirectional data format conversion. The upper-layer training framework always interacts with the training system 200 in a unified data format.

[0088] Figure 4 A flowchart of an example process 400 for data processing under certain circumstances is shown. Process 400 can be executed by the training system 200 described above. For ease of description, process 400 will be described below from the perspective of the training system 200. However, it should be understood that this is merely exemplary and is not intended to be limiting.

[0089] like Figure 4 As shown in block 410, the training system 200 uses a first operation interface among multiple operation interfaces to receive a first operation instruction conforming to a first data format. The first operation instruction is used to request a first environment instance among multiple environment instances to perform a first type of operation.

[0090] In box 420, the training system 200 uses a first adapter among a plurality of adapters to convert a first operation instruction into a second operation instruction conforming to a second data format, the first adapter corresponding to a first environment instance.

[0091] In box 430, the training system 200 uses a first environment interface among multiple environment interfaces to call a first environment instance based on a second operation instruction to perform a first type of operation in order to obtain a first execution result that conforms to a second data format.

[0092] In some cases, process 400 further includes: providing a first execution result to a first adapter through a first environment interface; converting the first execution result into a second execution result conforming to a first data format through the first adapter; and feeding back the second execution result for the first operation instruction through a first operation interface. In this way, the upper-layer caller can obtain the execution result in a unified data format without needing to worry about the differences in data formats of the underlying environment.

[0093] In some cases, receiving a first operation instruction via a first operation interface includes at least one of the following: receiving a reset instruction via a reset interface 211, the reset instruction being used to request that at least one of a plurality of environment instances be restored to an initial state, and to provide feedback on the first observation data of the at least one environment instance in the initial state; receiving a step instruction via a step interface 212, the step instruction carrying first action information, the step instruction instructing the at least one environment instance to perform a first action based on the first action information to update the environment state of the at least one environment instance, and to provide feedback on the feedback result related to the first action; receiving a rendering instruction via a rendering interface 213, the rendering instruction indicating a rendering mode, the rendering instruction being used to request visualization information of the at least one environment instance, the visualization information indicating the updated environment state of the at least one environment instance in the rendering mode; receiving a reward determination instruction via a reward determination interface 214, the reward determination instruction instructing the determination of first reward information corresponding to the updated environment state; receiving a termination determination instruction via a termination determination interface 215, the termination determination instruction instructing the determination of whether the task related to the first action is in a terminated state to obtain a determination result; and receiving a shutdown instruction via a shutdown interface 216, the shutdown instruction instructing the shutdown of the at least one environment instance and releasing the computing resources associated with the at least one environment instance.

[0094] In some cases, the feedback result includes at least one of the following: a first field indicating second observation data of the at least one environment instance, the second observation data indicating the updated environment state of the at least one environment instance; a second field indicating second reward information for the first action; a third field indicating whether the task related to the first action is in a terminated state; a fourth field indicating whether the task related to the first action is in a truncated state; and a fifth field indicating auxiliary information related to the at least one environment instance.

[0095] In some cases, multiple environment instances include at least one of the following: a first type of environment instance configured to provide a simulation environment for simulating a physical environment; a second type of environment instance configured to provide an execution environment for running tools; and a third type of environment instance configured to provide a verification environment for verifying the correctness of the inference results of a machine learning model.

[0096] In some cases, process 400 further includes: retrieving third observation information from observation queue 232 via an inference engine; obtaining third action information based on the third observation information using a machine learning model; and adding the third action information to action queue 231, wherein observation queue 232 is used to store observation information indicating the environmental state of at least one of a plurality of environmental instances, and action queue 231 is used to store action information indicating the execution of an action in the at least one environmental instance.

[0097] In some cases, process 400 further includes: retrieving second action information from action queue 231 via the environment execution engine; executing the second action corresponding to the second action information on at least one environment instance to update the environment state of the at least one environment instance; obtaining fourth observation information, which indicates the updated environment state of the at least one environment instance; and adding the fourth observation information to observation queue 232. In this way, policy reasoning and environment execution are decoupled via action queue 231 and observation queue 232, forming an asynchronous pipeline, eliminating synchronous waiting, and improving the utilization of computing resources.

[0098] In some cases, process 400 also includes: determining the third reward information corresponding to the second action through the environment execution engine; and saving the third reward information to the reward queue 233.

[0099] In some cases, process 400 further includes: aggregating the third action information of the first item through the first controller 234 and adding the third action information of the first item to the action queue 231; or aggregating the fourth observation information of the second item through the first controller 234 and adding the fourth observation information of the second item to the observation queue 232.

[0100] In some cases, process 400 further includes: determining a timeout threshold corresponding to the at least one environment instance based on the environment type of the at least one environment instance by a second controller 235; and performing timeout processing in response to the at least one environment instance performing a second action for a duration exceeding the timeout threshold, wherein the timeout thresholds corresponding to different environment types are different.

[0101] In some cases, process 400 further includes: determining the fill rate of the target queue via a third controller 236; and controlling the rate at which data is added to the target queue based on the fill rate, the target queue including at least one of an action queue 231 and an observation queue 232.

[0102] In some cases, controlling the rate at which data is added to the directional target queue based on the fill rate includes: reducing the rate at which data is added to the directional target queue in response to the fill rate exceeding a first threshold; and pausing the rate at which data is added to the directional target queue in response to the fill rate exceeding a second threshold, wherein the second threshold is greater than the first threshold.

[0103] In some cases, process 400 further includes at least one of the following: creating an environment instance corresponding to the configuration information using an environment execution engine based on the configuration information; resetting the environment instance corresponding to the first identification information using an environment execution engine based on a first identification information indicating the environment instance; performing actions in a set of environment instances corresponding to a set of identification information using an environment execution engine based on the set of action information to update the environment state of the set of environment instances; obtaining a set of observation information from the set of environment instances; and closing the environment instance corresponding to the second identification information based on a second identification information indicating the environment instance, and releasing the computing resources occupied by the environment instance.

[0104] In some cases, process 400 further includes: maintaining an environment preheating pool via a first manager 244, the environment preheating pool including at least one pre-created environment instance; and managing the at least one environment instance via a second manager 245.

[0105] In some cases, managing the at least one environment instance through the second manager 245 includes managing the at least one environment instance based on its running status, which includes the status as described above. Figure 3 At least one of the described preheating state 310, active state 320, ineffective state 330, recycling state 340 and destruction state 350.

[0106] In some cases, the multiple adapters also include a combination adapter 225, and the process 400 further includes: using the combination adapter 225 to perform a task using at least two of the multiple environment instances.

[0107] In some cases, performing a task using the at least two environment instances includes at least one of the following: in a first mode, performing actions in parallel in the at least two environment instances to update the environment state of the at least two environment instances, obtaining at least two fifth observation information items corresponding to the at least two environment instances, and obtaining sixth observation information by aggregating the at least two fifth observation information items; and in a second mode, performing actions sequentially in each environment instance according to a predetermined order based on the at least two action information items corresponding to the at least two environment instances, and using the observation information of the previous environment instance in the predetermined order to determine the action information of the next environment instance.

[0108] Through the above process 400, various types of environment instances can be accessed and invoked in a unified manner, shielding the underlying differences between heterogeneous environments. New environment access does not require modification of the core code of the training framework. At the same time, combined with mechanisms such as asynchronous interaction protocol layer, environment warm-up pool, state machine lifecycle management, environment combination and resource adaptive scheduling, it can significantly improve the utilization rate of computing resources in the training pipeline, shorten the cold start time of environment instances, provide automatic fault recovery capabilities, and meet the needs of complex multi-environment training scenarios.

[0109] Figure 5 A block diagram of an electronic device 500 in which one or more examples may be implemented is shown. It should be understood that... Figure 5 The electronic device 500 shown is merely exemplary and should not be construed as limiting the functionality and scope of the examples described herein. Figure 5 The electronic device 500 shown can be used to implement the training system discussed above.

[0110] like Figure 5 As shown, electronic device 500 is in the form of a general-purpose electronic device. Components of electronic device 500 may include, but are not limited to, one or more processing units or processors 510, memory 520, storage devices 530, one or more communication units 540, one or more input devices 550, and one or more output devices 530. Processor 510 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 520. In a multiprocessor system, multiple processors execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 500.

[0111] Electronic device 500 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 500, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 520 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof). Storage device 530 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data and can be accessed within electronic device 500.

[0112] Electronic device 500 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 5 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 520 may include computer program product 525 having one or more program modules configured to perform various methods or actions of various examples.

[0113] The communication unit 540 enables communication with other electronic devices via a communication medium. Additionally, the functionality of the components of the electronic device 500 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 500 can operate in a networked environment using logical connections to one or more other servers, networked personal computers, or another network node.

[0114] Input device 550 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 530 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 500 can also communicate with one or more external devices (not shown) via communication unit 540 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 500, or with any device that enables electronic device 500 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via an input / output (I / O) interface (not shown).

[0115] A computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. A computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0116] The flowcharts and / or block diagrams of the methods, apparatus, devices, and computer program products referred to herein describe various aspects. It should be understood that each block of the flowcharts and / or block diagrams, as well as combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0117] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0118] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0119] The flowcharts and block diagrams in the accompanying figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products under various scenarios. In this respect, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the figures. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0120] Various examples have been described above. The foregoing descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A system for data processing, comprising: The first layer provides multiple operation interfaces for supporting multiple types of interactive operations. The first layer is configured to use a first operation interface among the multiple operation interfaces to receive a first operation instruction conforming to a first data format. The first operation instruction is used to request a first environment instance among multiple environment instances to perform a first type of operation. The second layer provides multiple adapters corresponding to the plurality of environment instances. The second layer is configured to use a first adapter to convert the first operation instruction into a second operation instruction conforming to a second data format. The first adapter is the adapter among the plurality of adapters that corresponds to the first environment instance. as well as The third layer provides multiple environment interfaces corresponding to the multiple environment instances. The third layer is configured to use the first environment interface to call the first environment instance based on the second operation instruction to execute the first type of operation, so as to obtain a first execution result that conforms to the second data format.

2. The system of claim 1, wherein the third layer is further configured to provide the first execution result to the first adapter using the first environment interface; The second layer is further configured to utilize the first adapter to convert the first execution result into a second execution result conforming to the first data format; and The first layer is also configured to use the first operation interface to feed back the second execution result in response to the first operation instruction.

3. The system according to claim 1, wherein the plurality of operation interfaces includes at least one of the following: The reset interface is configured to receive a reset instruction, which requests that at least one of the plurality of environment instances be restored to an initial state and provides feedback the first observation data of the at least one environment instance in the initial state. A stepping interface is configured to receive a stepping instruction carrying first action information, the stepping instruction instructing the execution of a first action based on the first action information in the at least one environment instance to update the environment state of the at least one environment instance, and to provide feedback results related to the first action. A rendering interface is configured to receive rendering instructions, the rendering instructions indicating a rendering mode, the rendering instructions being used to request visualization information of the at least one environment instance, the visualization information indicating the updated environment state of the at least one environment instance in the rendering mode; The reward determination interface is configured to receive a reward determination instruction, which instructs the determination of first reward information corresponding to the updated environment state. The termination determination interface is configured to receive a termination determination instruction, which instructs to determine whether the task related to the first action is in a terminated state in order to obtain a determination result. The shutdown interface is configured to receive a shutdown command that instructs the shutdown of the at least one environment instance and releases the computing resources associated with the at least one environment instance.

4. The system according to claim 3, wherein the feedback result includes at least one of the following: The first field indicates the second observation data of the at least one environment instance, the second observation data indicating the updated environment state of the at least one environment instance; The second field indicates the second reward information for the first action; The third field indicates whether the task related to the first action is in a terminated state; The fourth field indicates whether the task related to the first action is in a truncated state; as well as The fifth field indicates auxiliary information related to the at least one environment instance.

5. The system of claim 1, wherein the plurality of environment instances includes at least one of the following: The first type of environment instance is configured to provide a simulation environment for simulating a physical environment; The second type of environment instance is configured to provide an execution environment for running tools; The third type of environment instance is configured to provide a verification environment for verifying the correctness of the inference results of the machine learning model.

6. The system according to claim 1, further comprising: An action queue is used to store second action information, which instructs at least one of the plurality of environment instances to perform a second action. An observation queue is used to store third observation information, which indicates the environmental state of the at least one environment instance. An inference engine is configured to retrieve the third observation information from the observation queue, obtain third action information based on the third observation information using a machine learning model, and add the third action information to the action queue, wherein the third action information indicates that a third action should be performed on at least one environment instance. An environment execution engine is configured to retrieve the second action information from the action queue, execute the second action on the at least one environment instance to update the environment state of the at least one environment instance, obtain fourth observation information, add the fourth observation information to the observation queue, and the fourth observation information indicates the updated environment state of the at least one environment instance.

7. The system of claim 6, wherein the environment execution engine is further configured to determine third reward information corresponding to the second action; and the system further comprises: The reward queue is configured to store the third reward information.

8. The system according to claim 6, further comprising: A first controller is configured to aggregate third action information of a first number of items and add the third action information of the first number of items to the action queue, or aggregate fourth observation information of a second number of items and add the fourth observation information of the second number of items to the observation queue.

9. The system according to claim 6, further comprising: The second controller is configured to determine a timeout threshold corresponding to the at least one environment instance based on the environment type of the at least one environment instance, and to perform timeout processing in response to the at least one environment instance performing the second action for a duration exceeding the timeout threshold, wherein the timeout thresholds corresponding to different environment types are different.

10. The system according to claim 6, further comprising: A third controller is configured to determine the fill rate of a target queue and, based on the fill rate, control the rate at which data is added to the target queue in the direction corresponding to the target queue, the target queue including at least one of the action queue and the observation queue.

11. The system of claim 10, wherein the third controller is further configured to: In response to the fill rate exceeding a first threshold, the rate at which data is added to the target queue in the data supply direction is reduced; and In response to the fill rate exceeding a second threshold, the data supply direction for adding data to the target queue is paused, wherein the second threshold is greater than the first threshold.

12. The system according to claim 1, further comprising: The environment execution engine is configured to manage the multiple environment instances, and The third layer also includes at least one of the following: The first engine interface is configured to create an environment instance corresponding to the configuration information using the environment execution engine based on the configuration information. The second engine interface is configured to reset the environment instance corresponding to the first identification information based on the first identification information of the environment instance using the environment execution engine. The third engine interface is configured to use the environment execution engine to perform actions in a set of environment instances corresponding to the set of identification information based on a set of identification information and a set of action information, so as to update the environment state of the set of environment instances. The fourth engine interface is configured to obtain a set of observation information from the set of environment instances; The fifth engine interface is configured to shut down the environment instance corresponding to the second identification information based on the second identification information of the environment instance, and release the computing resources occupied by the environment instance.

13. The system according to claim 1, further comprising: The first manager is configured to maintain an environment preheating pool, which includes at least one pre-created environment instance; as well as The second manager is configured to manage the at least one environment instance.

14. The system of claim 13, wherein the second manager is further configured to manage the at least one environment instance based on the running state of the at least one environment instance, the running state including at least one of the following: The preheating state indicates that the corresponding environment instance has been initialized and is in a state of waiting to be allocated. The active status indicates that the corresponding environment instance has been assigned to the first task; Failure status indicates that the corresponding environment instance is in an abnormal state; The recycling status indicates that the computing resources associated with the corresponding environment instance are in the process of being cleaned up; The destroyed status indicates that the computing resources associated with the corresponding environment instance have been released.

15. The system of claim 1, wherein the plurality of adapters further comprises: The combined adapter is configured to perform tasks using at least two of the plurality of environment instances.

16. The system of claim 15, wherein the combination adapter is further configured to perform at least one of the following: In the first mode, actions are performed in parallel in the at least two environment instances to update the environment state of the at least two environment instances, obtain at least two fifth observation information items corresponding to the at least two environment instances, and obtain sixth observation information by aggregating the at least two fifth observation information items. In the second mode, based on at least two action information corresponding to the at least two environment instances, actions are performed sequentially in each environment instance according to a predetermined order of the at least two environment instances, and the observation information of the previous environment instance in the predetermined order is used to determine the action information of the next environment instance.

17. A method for data processing, performed by the system of any one of claims 1 to 16, the method comprising: The first operation interface among multiple operation interfaces is used to receive a first operation instruction conforming to a first data format. The first operation instruction is used to request the first environment instance among multiple environment instances to perform a first type of operation. The first operation instruction is converted into a second operation instruction conforming to a second data format using a first adapter among multiple adapters, and the first adapter corresponds to the first environment instance; as well as The first environment interface among multiple environment interfaces is used to call the first environment instance based on the second operation instruction to execute the first type of operation, so as to obtain a first execution result that conforms to the second data format.

18. The method of claim 17, further comprising: The first execution result is provided to the first adapter through the first environment interface; The first adapter converts the first execution result into a second execution result that conforms to the first data format. as well as The first operation interface provides feedback on the second execution result in response to the first operation instruction.

19. The method of claim 17, wherein the plurality of environment instances comprises at least one of the following: The first type of environment instance is configured to provide a simulation environment for simulating a physical environment; The second type of environment instance is configured to provide an execution environment for running tools; The third type of environment instance is configured to provide a verification environment for verifying the correctness of the inference results of the machine learning model.

20. The method of claim 17, further comprising: The third observation information is retrieved from the observation queue using the inference engine; The third action information is obtained based on the third observation information using a machine learning model; as well as The third action information is added to the action queue, wherein the observation queue is used to store observation information indicating the environmental state of at least one of the plurality of environmental instances, and the action queue is used to store action information indicating the execution of an action in the at least one environmental instance.

21. The method of claim 20, further comprising: The second action information is retrieved from the action queue by the environment execution engine; The at least one environment instance performs a second action corresponding to the second action information to update the environment state of the at least one environment instance. Obtain fourth observation information, which indicates the updated environmental state of the at least one environment instance; and The fourth observation information is added to the observation queue.

22. An electronic device, comprising: At least one processing unit; as well as At least one memory, coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, which, when executed by the at least one processing unit, cause the electronic device to perform the method according to any one of claims 17 to 21.

23. A computer-readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement the method according to any one of claims 17 to 21.

24. A computer program product tangibly stored in a computer storage medium and comprising computer-executable instructions that, when executed by a device, cause the device to perform the method according to any one of claims 17 to 21.

Images