Data processing method based on reinforcement learning, electronic device and readable medium

Abstract

Embodiments of the present disclosure disclose a data processing method based on reinforcement learning, an electronic device and a readable medium. The method is applied to an execution node, a policy node and a training node included in a reinforcement learning system, and a specific implementation of the method comprises: performing environment simulation by the execution node to obtain an observation result; performing policy updating by the training node according to a training sample and a model to be trained, wherein the training sample is transmitted from the execution node to the training node through a sample transmission stream; the policy node performs policy inference according to the observation result and updated policy information, wherein in a non-in-line inference mode, the execution node and the policy node perform data transmission on the observation result and an action to be executed through an inference transmission stream; and the execution node executes the action to be executed to generate an updated observation result. The implementation realizes effective scheduling of large-scale computer hardware resources, meets the training requirements for complex reinforcement learning tasks, and improves the training speed.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the field of computer technology, and more particularly to data processing methods, electronic devices, and readable media based on reinforcement learning. Background Technology

[0002] Reinforcement learning (RL) is an important subfield of machine learning, used to train a model or agent to learn policies during interaction with its environment, aiming to maximize rewards. As the complexity of reinforcement learning tasks continues to increase, efficiently processing large amounts of data for training the model or agent becomes crucial. Currently, the common approach to reinforcement learning is to train the model or agent using small to medium-sized computer hardware resources.

[0003] However, the inventors discovered that the following technical problems often arise when using the above method:

[0004] First, small- to medium-sized computer hardware resources are insufficient to effectively support complex reinforcement learning tasks, resulting in slow training speed.

[0005] Second, existing reinforcement learning systems lack support for large-scale distributed computer hardware resources, making it impossible to meet the training requirements of complex reinforcement learning tasks.

[0006] Third, in conventional reinforcement learning, "environment simulation-policy reasoning" is a basic step unit. When policy reasoning is performed, computer hardware resources will be idle periodically, resulting in low efficiency of computer hardware resource utilization.

[0007] The information disclosed in this background section is only intended to enhance the understanding of the background of the inventive concept, and therefore may contain information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0008] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0009] Some embodiments of this disclosure propose data processing methods, electronic devices, and readable media based on reinforcement learning to solve one or more of the technical problems mentioned in the background section above.

[0010] In a first aspect, some embodiments of this disclosure provide a data processing method based on reinforcement learning, which is applied to an execution node, a policy node, and a training node in a reinforcement learning system. The method includes: performing environment simulation through the execution node to obtain observation results; updating the policy through the training node based on training samples and a model to be trained to generate updated policy information, wherein the training samples are transmitted from the execution node to the training node via a sample transport stream; performing policy inference through the policy node based on the observation results and the updated policy information to generate an action to be executed, wherein, in a non-inline inference mode, the execution node and the policy node transmit the observation results and the action to be executed via an inference transport stream; and executing the action to be executed through the execution node to generate updated observation results.

[0011] Secondly, some embodiments of this disclosure provide a data processing apparatus based on reinforcement learning, applied to an execution node, policy node, and training node in a reinforcement learning system. The apparatus includes: an environment simulation unit configured to perform environment simulation through the execution node to obtain observation results; a policy update unit configured to update the policy through the training node based on training samples and a model to be trained to generate updated policy information, wherein the training samples are transmitted from the execution node to the training node via a sample transport stream; a policy inference unit configured to perform policy inference by the policy node based on the observation results and the updated policy information to generate an action to be executed, wherein, in non-inline inference mode, the execution node and the policy node transmit the observation results and the action to be executed via an inference transport stream; and an execution unit configured to execute the action to be executed by the execution node to generate updated observation results.

[0012] Thirdly, some embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation of the first aspect above.

[0013] Fourthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method described in any of the implementations of the first aspect above.

[0014] The above embodiments of this disclosure have the following beneficial effects: Through the data processing methods based on reinforcement learning in some embodiments of this disclosure, effective scheduling of large-scale computer hardware resources is achieved, the training requirements for complex reinforcement learning tasks are met, and training speed is improved. Specifically, the reasons for the inability to meet the training requirements of complex reinforcement learning tasks and the low training speed are: First, small-to-medium-scale computer hardware resources are insufficient to effectively support complex reinforcement learning tasks, resulting in low training speed; Second, existing reinforcement learning systems lack support for large-scale distributed computer hardware resources, making it impossible to meet the training requirements of complex reinforcement learning tasks. In practice, due to the limited analysis of complex reinforcement learning tasks in the past, there is a lack of effective ways to optimize performance for large-scale computer hardware resources, making it difficult for common open-source reinforcement learning systems to support complex reinforcement learning tasks. Based on this, firstly, this disclosure decouples the "environment simulation-policy update" process, that is, by constructing three basic nodes: an execution node, a policy node, and a training node, to respectively realize environment simulation, policy inference, and policy update. This solves the problem that common reinforcement learning systems can only train with "environment simulation-policy update" as the smallest unit. Secondly, environmental simulation is performed through the aforementioned execution nodes to obtain observation results. Then, the policy is updated through the aforementioned training nodes based on the training samples and the model to be trained, generating updated policy information to achieve timely updates of policy information. Specifically, the aforementioned training samples are transmitted from the aforementioned execution nodes to the aforementioned training nodes via a sample transport stream. Further, the aforementioned policy nodes perform policy inference based on the aforementioned observation results and the aforementioned updated policy information to generate actions to be executed. In non-inline inference mode, the aforementioned execution nodes and the aforementioned policy nodes transmit data between the observation results and the actions to be executed via an inference transport stream. Finally, the aforementioned execution nodes execute the actions to be executed to generate updated observation results. It can be observed that, firstly, by abstracting the reinforcement learning task into execution nodes, policy nodes, and training nodes, reinforcement learning tasks can be easily deployed on computer hardware resources. Simultaneously, it also possesses good scalability in terms of computer hardware resource expansion, enabling reasonable and flexible expansion of corresponding computer hardware resources according to the complexity of the reinforcement learning task. Furthermore, by abstracting the communication methods between nodes into sample transport streams and inference transport streams, efficient data transmission between different nodes is achieved. In summary, the data processing method based on reinforcement learning disclosed in this paper improves the support of computer hardware resources for complex reinforcement learning tasks and meets the requirements for efficient and high-speed training. Attached Figure Description

[0015] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0016] Figure 1 This is a schematic diagram illustrating an application scenario of a data processing method based on reinforcement learning, which is one of the embodiments of this disclosure.

[0017] Figure 2 This is a flowchart of some embodiments of the data processing method based on reinforcement learning according to the present disclosure;

[0018] Figure 3 This is a diagram illustrating the reinforcement learning process;

[0019] Figure 4 This is a schematic diagram illustrating the process of policy reasoning based on environmental instance rings;

[0020] Figure 5 This is a schematic diagram illustrating the process by which the parameter server broadcasts updated policy information to the policy node in some embodiments of this disclosure;

[0021] Figure 6 This is a schematic diagram of the structure of some embodiments of the data processing apparatus based on reinforcement learning according to the present disclosure;

[0022] Figure 7 This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0023] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0024] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0025] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0026] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0027] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0028] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0029] Figure 1 This is a schematic diagram illustrating an application scenario of a data processing method based on reinforcement learning, which is one of the embodiments of this disclosure.

[0030] exist Figure 1 In this application scenario, firstly, the computing device performs environmental simulation through the execution node 101 to obtain observation results. Secondly, the computing device updates the policy based on the training samples and the model to be trained through the training node 103 to generate updated policy information. The training samples are transmitted from the execution node 101 to the training node 103 via the sample transport stream 104. Next, the computing device performs policy inference based on the observation results and the updated policy information through the policy node 102 to generate actions to be executed. In non-inline inference mode, the execution node 101 and the policy node 102 transmit the observation results and actions to be executed via the inference transport stream 105. Finally, the computing device executes the actions to be executed through the execution node 101 to generate updated observation results. In this application scenario, the number of execution nodes 101, policy nodes 102, and training nodes can be expanded as needed, for example, as... Figure 1 As shown, the reinforcement learning system may include: 2 execution nodes 101, 2 policy nodes 102 and 2 training nodes 103.

[0031] The computing device can be either hardware or software. When the computing device is software, it can be implemented as a reinforcement learning system consisting of execution nodes, policy nodes, and training nodes. When the computing device is hardware, it can be implemented as computer hardware resources that support the above reinforcement learning system, such as a distributed cluster composed of multiple servers. It should be noted that the number of execution nodes, policy nodes, and training nodes included in the reinforcement learning system can be arbitrarily expanded as needed, and is not limited here.

[0032] Continue to refer to Figure 2The diagram illustrates a flow 200 of some embodiments of a reinforcement learning-based data processing method according to this disclosure. This reinforcement learning-based data processing method is applied to an execution node, policy node, and training node in a reinforcement learning system. The execution node, policy node, and training node are all independently schedulable and are located on multiple machines with heterogeneous computer hardware resources. That is, the execution node, policy node, and training phase are all allocated computer hardware resources suitable for their processing tasks. The method includes the following steps:

[0033] Step 201: Perform environmental simulation by executing nodes and obtain observation results.

[0034] In some embodiments, the execution entity (e.g., a computing device) of a reinforcement learning-based data processing method can perform environmental simulation through execution nodes to obtain observation results. The execution nodes are used to perform environmental simulation to generate corresponding observation results. In practice, the execution nodes also generate reward signals during the environmental simulation process. The observation results are partial descriptive information of the state. The reward signal can be a signal value perceived from the environment during the environmental simulation process, used to characterize the quality of the state. Specifically, based on the reward signal, the cumulative reward can be maximized to obtain optimal policy information.

[0035] As an example, such as Figure 3 The diagram illustrates a reinforcement learning process in one round. First, the decision agent selects an action during the interaction. Then, environmental simulation is performed based on the action to generate corresponding observations and reward signals, which influence the decision agent's action selection in the next round. In this application, the environmental simulation task is executed by an execution node to generate observations and reward signals.

[0036] In some optional implementations of certain embodiments, the execution entity performs environmental simulation through the execution node to obtain observation results, which may include the following steps:

[0037] Based on the central processing unit, the aforementioned execution nodes perform environmental simulations using an externally imported black-box program to generate the aforementioned observations.

[0038] In this context, a black-box program can be a program whose internal structure is unknown, but whose inputs, outputs, and functions are known. In practice, a black-box program can be a game engine or a physics engine. Specifically, the execution nodes in this application rely on a central processing unit (CPU) for environment simulation. When computer hardware resources (CPU) are sufficient, the aforementioned execution entity can run multiple execution nodes in parallel to perform parallel environment simulation. When multiple execution nodes exist, the execution nodes within these nodes are independent of each other.

[0039] In some optional implementations of certain embodiments, the execution entity performs environmental simulation through the execution node to obtain observation results, which may include the following steps:

[0040] For each environment instance in the environment instance ring, in response to determining that the above policy node performs policy reasoning based on the observation results corresponding to the target environment instance, the above execution node performs environment simulation on the above environment instance.

[0041] In this context, at least one environment instance within the environment instance ring is sequentially invoked by the aforementioned execution nodes. The target environment instance is the environment instance within the environment instance ring that precedes the aforementioned environment instance. In practice, in object-oriented programming, an instance refers to the result of instantiating an object. The size of the environment instance ring can be adjusted; that is, the number of environment instances in the ring can be flexibly set as needed.

[0042] As an example, such as Figure 4 The diagram illustrates the policy reasoning process based on an environment instance ring, where the environment instance ring can include environment instance A, environment instance B, environment instance C, and environment instance D. The environment instances in the ring are invoked by the execution node in a clockwise order. For example, environment instance A is invoked before environment instance B. Specifically, taking environment instance B and environment instance C as examples, at time T, when environment instance B is invoked by execution node A to perform environment simulation and generate observation result B corresponding to environment instance B. Then, at time T+1, policy node A performs policy reasoning based on observation result B and updated policy information to generate action B to be executed. At this time (time T+1), execution node A is idle, so environment instance C can be loaded onto execution node A to perform environment simulation and generate observation result C.

[0043] The environment instance ring and its usage, as an inventive point of this disclosure, solves the third technical problem mentioned in the background art: in conventional reinforcement learning, "environment simulation-policy reasoning" is a basic step unit. When policy reasoning is performed, computer hardware resources are periodically idle, resulting in low efficiency of computer hardware resource utilization. As shown in the example above, by setting up an environment instance ring, the environment instances can be called sequentially, avoiding calling chaos. At the same time, it can be ensured that the execution node is not idle when the policy node is performing policy reasoning, that is, the computer hardware resources (CPU) corresponding to the execution node are not idle, thereby improving the efficiency of computer hardware resource utilization.

[0044] Step 202: Update the policy based on the training samples and the model to be trained through the training node to generate updated policy information.

[0045] In some embodiments, the aforementioned execution entity can update the policy based on training samples and the model to be trained through a training node to generate updated policy information. The training samples are transmitted from the execution node to the training node via a sample transport stream. These training samples are used for policy updates. The sample transport stream can be a simplex data transmission stream between the execution node and the training node. Specifically, the sample transport stream can transmit training samples unidirectionally between the execution node and the training node via streaming sockets. The model to be trained can be a machine learning model to be trained using reinforcement learning. In practice, to avoid a single training node becoming a bottleneck in the entire reinforcement learning system, multiple training nodes can run in parallel. That is, for multiple training nodes, the training samples can be evenly distributed among them, with each training node containing the same model to be trained and performing gradient calculations independently. Specifically, a single training node may not be able to fully utilize the GPU; therefore, multiple training nodes in the reinforcement learning system of this application can share the GPU.

[0046] In some optional implementations of certain embodiments, the execution entity updates the policy based on the training samples and the model to be trained to generate updated policy information, which may include the following steps:

[0047] The first step is to load the above training samples into the cache.

[0048] In practice, the aforementioned execution entity can load training samples into the cache of the server.

[0049] The second step is to load the training samples from the cache into the graphics processor.

[0050] Training nodes can be assigned graphics processing units (GPUs) for policy updates. In practice, the GPUs assigned to different training nodes can be heterogeneous.

[0051] Third, the training nodes update the policy using gradient descent based on the training samples and the model to be trained loaded into the graphics processor, thereby generating the updated policy information.

[0052] In practice, receiving and loading data are I / O-intensive operations. Overlapping the first and second steps with the third step may result in higher sample throughput, but model updates (policy updates) in batch processing of training samples depend on data loading. Therefore, this application designs a data prefetching technique, that is, storing two batches of training samples (the first batch of training samples and the second batch of training samples) in GPU memory. When training starts, the first batch of training samples is loaded into GPU memory. When the GPU calculates the gradient corresponding to the first batch of training samples, the third batch of training samples is stored in GPU memory. When the GPU calculates the gradient corresponding to the second batch of training samples, the fourth batch of training samples is loaded into GPU memory. This allows the training node to execute the first to third steps in parallel threads, improving training performance.

[0053] In some optional implementations of certain embodiments, after updating the policy based on the training samples and the model to be trained through the training node to generate updated policy information, the method further includes:

[0054] The first step is to push the updated strategy information to the parameter database for storage.

[0055] The parameter database can be a database used to store the policy parameters corresponding to the policy information. In practice, the parameter database can be set up on a parameter server.

[0056] The second step is to periodically broadcast the updated policy information stored in the parameter database to the policy nodes via the parameter server.

[0057] In practice, the aforementioned execution entity can provide parameter update services by running a service thread on NFS (Network File System) or on a node (e.g., a parameter server). Specifically, running a service thread on a node offers slightly better performance than NFS, but NFS has higher data throughput.

[0058] As an example, such as Figure 5The diagram shown illustrates the process by which the parameter server broadcasts updated policy information to the policy nodes. The parameter server 501 can periodically send the updated policy information to policy nodes A5021, B5022, ..., L5023 and M5024 in the form of broadcast.

[0059] In some optional implementations of certain embodiments, after periodically broadcasting the updated policy information stored in the parameter database to the policy node via the parameter server, the method further includes:

[0060] The first step is to determine the version information corresponding to the updated policy information through the aforementioned policy nodes.

[0061] The version information represents the version of the policy information. In practice, the policy node can determine the version information corresponding to the updated policy information stored in the parameter database. Alternatively, the policy node can send a request to the parameter server to query the version information corresponding to the updated policy information stored in the parameter database.

[0062] The second step is to determine that the version of the updated policy information is the target version, and then retrieve the updated policy information from the parameter server through the policy node.

[0063] In practice, the target version mentioned above can be the latest version.

[0064] Step 203: The policy node performs policy reasoning based on the observation results and the updated policy information to generate actions to be executed.

[0065] In some embodiments, the aforementioned executing entity can perform policy reasoning based on observations and updated policy information through policy nodes to generate actions to be executed. These actions can be actions to be executed in the next stage. In practice, policy nodes can perform batch policy reasoning.

[0066] In some optional implementations of certain embodiments, the aforementioned execution entity performs policy reasoning based on observation results and updated policy information through policy nodes to generate actions to be executed, which may include:

[0067] The aforementioned policy nodes, through a central processing unit or graphics processing unit, perform policy reasoning using a forward propagation method based on the aforementioned observations and the aforementioned updated policy information, in order to generate the aforementioned actions to be executed.

[0068] In some optional implementations of certain embodiments, the execution entity, through a policy node, performs policy reasoning via a central processing unit or graphics processing unit, based on the observed results and the updated policy information, using a forward propagation approach to generate the action to be executed. This may include the following steps:

[0069] In response to the above policy node not being allocated a graphics processor or the network bandwidth value being less than the network bandwidth threshold, inline inference mode is enabled, and the following processing steps are executed:

[0070] The first step is to start the policy node locally.

[0071] In practice, the above policy node can be started on a local server.

[0072] The second step, in response to successful startup, involves the central processing unit performing policy reasoning through forward propagation based on the aforementioned observations and updated policy information, in order to generate the aforementioned actions to be executed.

[0073] In the aforementioned inline inference mode, the execution node and the policy node transmit observations and actions to be executed via memory. That is, the execution node and policy node transmit data directly through memory, without using an inference transport stream, and transmit data via network. Specifically, in inline inference mode, data transmission between the execution node and policy node can be achieved through shared memory.

[0074] Step 204: The execution node performs the action to be executed to generate the updated observation results.

[0075] In some embodiments, the aforementioned execution entity can execute the action to be executed through the execution node to generate updated observation results. In practice, the aforementioned execution node can perform environmental simulation while executing the aforementioned action to be executed to generate updated observation results.

[0076] Optionally, the aforementioned execution entity may repeat steps 201 to 204 until the interaction ends, in order to achieve multiple rounds of reinforcement learning training.

[0077] The above embodiments of this disclosure have the following beneficial effects: Through the data processing methods based on reinforcement learning in some embodiments of this disclosure, effective scheduling of large-scale computer hardware resources is achieved, the training requirements for complex reinforcement learning tasks are met, and training speed is improved. Specifically, the reasons for the inability to meet the training requirements of complex reinforcement learning tasks and the low training speed are: First, small-to-medium-scale computer hardware resources are insufficient to effectively support complex reinforcement learning tasks, resulting in low training speed; Second, existing reinforcement learning systems lack support for large-scale distributed computer hardware resources, making it impossible to meet the training requirements of complex reinforcement learning tasks. In practice, due to the limited analysis of complex reinforcement learning tasks in the past, there is a lack of effective ways to optimize performance for large-scale computer hardware resources, making it difficult for common open-source reinforcement learning systems to support complex reinforcement learning tasks. Based on this, firstly, this disclosure decouples the "environment simulation-policy update" process, that is, by constructing three basic nodes: an execution node, a policy node, and a training node, to respectively realize environment simulation, policy inference, and policy update. This solves the problem that common reinforcement learning systems can only train with "environment simulation-policy update" as the smallest unit. Secondly, environmental simulation is performed through the aforementioned execution nodes to obtain observation results. Then, the policy is updated through the aforementioned training nodes based on the training samples and the model to be trained, generating updated policy information to achieve timely updates of policy information. Specifically, the aforementioned training samples are transmitted from the aforementioned execution nodes to the aforementioned training nodes via a sample transport stream. Further, the aforementioned policy nodes perform policy inference based on the aforementioned observation results and the aforementioned updated policy information to generate actions to be executed. In non-inline inference mode, the aforementioned execution nodes and the aforementioned policy nodes transmit data between the observation results and the actions to be executed via an inference transport stream. Finally, the aforementioned execution nodes execute the actions to be executed to generate updated observation results. It can be observed that, firstly, by abstracting the reinforcement learning task into execution nodes, policy nodes, and training nodes, reinforcement learning tasks can be easily deployed on computer hardware resources. Simultaneously, it also possesses good scalability in terms of computer hardware resource expansion, enabling reasonable and flexible expansion of corresponding computer hardware resources according to the complexity of the reinforcement learning task. Furthermore, by abstracting the communication methods between nodes into sample transport streams and inference transport streams, efficient data transmission between different nodes is achieved. In summary, the data processing method based on reinforcement learning disclosed in this paper improves the support of computer hardware resources for complex reinforcement learning tasks and meets the requirements for efficient and high-speed training.

[0078] Furthermore, for a better understanding of the beneficial effects of this application, please refer to the following:

[0079] First, we will analyze each existing reinforcement learning system individually:

[0080] (1) Vanilla reinforcement learning system: Its system design is strongly coupled, that is, "environment simulation-policy reasoning" is used as a basic step unit, and environment simulation and policy reasoning are performed alternately in each iteration. In this mode, it can ensure that the training samples are generated based on the latest policy information. However, the efficiency is extremely low. Although multiple environment examples can be started at the same time for environment simulation, the efficiency problem still cannot be solved.

[0081] (2) RLlib reinforcement learning system: It uses a single CPU for environment simulation and policy inference. However, it only supports multi-GPU training on the local machine via Python threads and cannot be expanded to multiple nodes, resulting in poor scalability.

[0082] (3) SeedRL reinforcement learning system: It uses a single GPU for policy inference and training. It can only run on a single GPU.

[0083] Meanwhile, the RLlib and SeedRL reinforcement learning systems also have limitations for complex reinforcement learning tasks. Specifically, they cannot be scaled to multiple nodes and are constrained by GPU memory limitations, ultimately failing to effectively meet the training requirements of complex reinforcement learning tasks.

[0084] Further reference Figure 6 As an implementation of the methods shown in the above figures, this disclosure provides some embodiments of a data processing apparatus based on reinforcement learning, which are similar to... Figure 2 Corresponding to the method embodiments shown, this reinforcement learning-based data processing device can be specifically applied to various electronic devices.

[0085] like Figure 6As shown, a reinforcement learning-based data processing apparatus 600 in some embodiments includes: an environment simulation unit 601, a policy update unit 602, a policy inference unit 603, and an execution unit 604. The environment simulation unit 601 is configured to perform environment simulation through the execution node to obtain observation results; the policy update unit 602 is configured to update the policy through the training node based on training samples and a model to be trained, to generate updated policy information, wherein the training samples are transmitted from the execution node to the training node via a sample transport stream; the policy inference unit 603 is configured to perform policy inference based on the observation results and the updated policy information to generate an action to be executed, wherein in non-inline inference mode, the execution node and the policy node transmit the observation results and the action to be executed via an inference transport stream; the execution unit 604 is configured to execute the action to be executed by the execution node to generate updated observation results.

[0086] It is understandable that the units and references described in the reinforcement learning-based data processing device 600 Figure 2 The steps in the described method correspond accordingly. Therefore, the operations, features, and beneficial effects described above for the method also apply to the reinforcement learning-based data processing device 600 and the units contained therein, and will not be repeated here.

[0087] The following is for reference. Figure 7 It shows a schematic diagram of the structure of an electronic device (e.g., a computing device) 700 suitable for implementing some embodiments of the present disclosure. Figure 7 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0088] like Figure 7 As shown, the electronic device 700 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory 702 or a program loaded from a storage device 708 into a random access memory 703. The random access memory 703 also stores various programs and data required for the operation of the electronic device 700. The processing unit 701, the read-only memory 702, and the random access memory 703 are interconnected via a bus 704. An input / output interface 705 is also connected to the bus 704.

[0089] Typically, the following devices can be connected to I / O interface 705: input devices 706 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 707 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 708 including, for example, magnetic tapes, hard disks, etc.; and communication devices 709. Communication device 709 allows electronic device 700 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 7 An electronic device 700 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 7 Each box shown can represent a device or multiple devices as needed.

[0090] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 709, or installed from a storage device 708, or installed from a read-only memory 702. When the computer program is executed by the processing device 701, it performs the functions defined in the methods of some embodiments of this disclosure.

[0091] It should be noted that, in some embodiments of this disclosure, the computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0092] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed networks.

[0093] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: perform environmental simulation through the aforementioned execution node to obtain observation results; perform policy updates through the aforementioned training node based on training samples and the model to be trained to generate updated policy information, wherein the aforementioned training samples are transmitted from the aforementioned execution node to the aforementioned training node via a sample transport stream; the aforementioned policy node performs policy inference based on the aforementioned observation results and the aforementioned updated policy information to generate an action to be executed, wherein, in non-inline inference mode, the aforementioned execution node and the aforementioned policy node transmit data between the observation results and the action to be executed via an inference transport stream; and the aforementioned execution node executes the action to be executed to generate updated observation results.

[0094] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0095] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0096] The units described in some embodiments of this disclosure can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor may be described as including an environment simulation unit, a policy update unit, a policy inference unit, and an execution unit. The names of these units do not necessarily limit the specific unit; for example, the environment simulation unit may also be described as "a unit that performs environment simulation through the aforementioned execution nodes to obtain observation results."

[0097] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0098] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A data processing method based on reinforcement learning, applied to the execution nodes, policy nodes, and training nodes of a reinforcement learning system, wherein, The execution node, the policy node, and the training node can all be independently scheduled and are located on multiple machines with heterogeneous computer hardware resources. The method includes: Environmental simulation is performed through the execution node to obtain observation results, including: For each environment instance in the environment instance ring, in response to determining that the policy node is performing policy reasoning based on the observation results corresponding to the target environment instance, the execution node performs environment simulation on the environment instance, wherein the environment instances in at least one environment instance included in the environment instance ring are sequentially invoked by the execution node, and the target environment instance is the environment instance in the environment instance ring that precedes the target environment instance; The training node updates the policy based on the training samples and the model to be trained to generate updated policy information. The training samples are transmitted from the execution node to the training node via a sample transport stream. The policy node, through a central processing unit or a graphics processing unit, performs policy reasoning in a forward propagation manner based on the observation results and the updated policy information to generate an action to be executed. In non-inline reasoning mode, the execution node and the policy node transmit the observation results and the action to be executed through an inference transport stream. The execution node performs the action to be performed in order to generate the updated observation results; The policy node, through a central processing unit or graphics processing unit, performs policy reasoning using a forward propagation approach based on the observation results and the updated policy information to generate actions to be executed, including: In response to the policy node not being allocated a graphics processor or the network bandwidth value being less than the network bandwidth threshold, inline inference mode is enabled, and the following processing steps are performed: Start the policy node locally; In response to successful startup, the central processing unit performs policy reasoning using forward propagation based on the observation results and the updated policy information to generate the action to be executed. In the inline reasoning mode, the execution node and the policy node transmit the observation results and the action to be executed through memory.

2. The method according to claim 1, wherein, After updating the policy based on the training samples and the model to be trained through the training node to generate updated policy information, the method further includes: The updated strategy information is pushed to the parameter database for storage; The updated policy information stored in the parameter database is periodically broadcast to the policy node via the parameter server.

3. The method according to claim 2, wherein, The environmental simulation performed through the execution node to obtain observation results includes: Based on the central processing unit, the execution node performs environmental simulation through an externally imported black-box program to generate the observation results.

4. The method according to claim 2, wherein, The step of updating the policy based on the training samples and the model to be trained through the training node to generate updated policy information includes: Load the training samples into the cache; Load the training samples loaded in the cache into the graphics processor; The training node updates its policy using gradient descent based on the training samples and the model to be trained loaded into the graphics processor, thereby generating the updated policy information.

5. The method according to claim 2, wherein, After periodically broadcasting the updated policy information stored in the parameter database to the policy node via the parameter server, the method further includes: The version information corresponding to the updated policy information is determined through the policy node; In response to determining that the version information represents the version of the updated policy information as the target version, the updated policy information is retrieved from the parameter server through the policy node.

6. An electronic device, comprising: One or more processors; A storage device on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 5.

7. A computer-readable medium having a computer program stored thereon, wherein, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 5.

Images