Reinforcement learning data sampling method, system, device and storage medium

Abstract

The application relates to a reinforcement learning data sampling method, system, device and storage medium. An environment state sample generated by an environment interaction node is acquired by a center node, the environment state sample is stored in an experience set, the environment state sample is sent to an action calculation node, an action sample generated by the action calculation node is received and stored, the action sample is sent to the environment interaction node, the environment state sample updated by the environment interaction node according to the action sample is continuously received and processed, a trajectory sample is generated according to the environment state sample and the action sample, when the number of trajectory samples stored in the experience set reaches a first threshold value, the trajectory sample is sent to a policy update node, the model is optimized according to the trajectory sample, and the optimized model parameter is sent to the action calculation node, efficient reading and writing of various node information are ensured, and the problem that the sampling efficiency of reinforcement learning is low in the prior art under the condition that the computing resources are limited is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of reinforcement learning data sampling of reinforcement learning, in particular to a reinforcement learning data sampling method, system, device and storage medium. BACKGROUND

[0002] Reinforcement learning is a type of learning problem in the field of machine learning. The main difference between reinforcement learning and common supervised learning and unsupervised learning is that reinforcement learning learns through interaction and feedback with the environment. Reinforcement learning is closer to the learning process of a living being, more intelligent, and closer to "strong artificial intelligence" than supervised learning or unsupervised learning. In recent years, reinforcement learning has achieved remarkable results in many fields such as video games, robot control, natural language processing, etc. In the training process of reinforcement learning, a large number of training samples are often required. Therefore, how to improve the sample collection efficiency in the training process of reinforcement learning has become an important direction of current reinforcement learning system research.

[0003] In the related art, common reinforcement learning frameworks include Google's SEEDRL, OpenAI's FIVE, and UCB's RLlib. However, these existing frameworks have different problems for ordinary researchers. SEEDRL uses multiple CPUs (central processing units) for parallel sampling and sends the collected sample data to a high-performance central server for training. In order to ensure training efficiency, the central server uses TPU (tensor processing unit) for forward inference and training. For ordinary researchers, this method inevitably encounters performance limitations when used. OpenAI FIVE abstracts the model's forward inference and backward propagation to different nodes in order to ensure as much parallelism as possible. Samples are transmitted between sampling nodes, forward inference nodes, and backward propagation nodes (training nodes). This approach can improve the utilization of computing resources, but it greatly increases network transmission costs and system control difficulty, limiting the improvement of sampling efficiency in the case of limited computing resources for ordinary researchers. RLlib is a general reinforcement learning library designed for researchers to quickly implement reinforcement learning algorithms. However, it does not optimize the efficiency of parallel sampling. Experiments show that its sampling efficiency is somewhat inferior to other frameworks. In addition, the above-mentioned related technologies also have a common problem, which is that the balance between sampling efficiency and training efficiency usually needs to be set manually, which often leads to uneven use of computing resources, resulting in imbalance between sampling efficiency and training efficiency.

[0004] Currently, there is no effective solution to the problem of low sampling efficiency of reinforcement learning in the related art under the condition of limited computing resources. SUMMARY

[0005] Embodiments of the present application provide a reinforcement learning data sampling method, system, device and storage medium to at least solve the problem of low sampling efficiency of reinforcement learning in the related art under limited computing resources.

[0006] In a first aspect, embodiments of the present application provide a reinforcement learning data sampling method, characterized in that comprising:

[0007] obtaining an environment state sample generated by an environment interaction node, and storing the environment state sample in an experience set of a center node;

[0008] sending the environment state sample to an action calculation node, receiving an action sample generated by the action calculation node according to a model parameter and the environment state sample, and storing the action sample in the experience set;

[0009] sending the action sample to the environment interaction node, continuing to receive and process the environment state sample updated by the environment interaction node according to the action sample; generating a trajectory sample according to the environment state sample and the corresponding action sample in the experience set, and when the number of trajectory samples stored in the experience set reaches a first threshold, sending the trajectory sample to a policy update node, wherein the policy update node optimizes a model according to the trajectory sample and outputs the optimized model parameter to the action calculation node, and the first threshold is dynamically adjusted according to model complexity, model convergence speed and other model indicators.

[0010] In some embodiments, the center node includes a high-speed read-write module, which is configured to create a session between the center node and a target node, and perform read-write operations on sample data based on the session, wherein the target node includes the environment interaction node, the action calculation node or the policy update node, and the sample data includes the environment state sample, the action sample or the trajectory sample.

[0011] In some embodiments, the method further comprises:

[0012] The high-speed read-write module page-encapsulates a storage space of the experience set, and each page of the storage space corresponds to different address information;

[0013] The high-speed read-write module performs parallel read-write on different pages of the storage space.

[0014] In some embodiments, the high-speed read-write module further includes a page controller configured to determine whether the address information is available, and find page storage information corresponding to the address information according to the address information.

[0015] In some embodiments, when the environment interaction node connects with the center node, the method comprises:

[0016] The high-speed read-write module creates a first session when connecting with the environment interaction node for the first time, processes the connection task of the environment interaction node with the center node based on the first session, and is used to save first address information;

[0017] The page controller receives and judges whether the first address information sent by the first session is available, and in the case that the first address information is available, locates the current first sample record according to the first address information, writes the current first sample record into the environment state sample, updates the first address information, and sends the updated first address information to the environment interaction node.

[0018] In some embodiments, when the action calculation node connects with the center node, the method comprises:

[0019] The high-speed read-write module creates a second session when connecting with the action calculation node for the first time, processes the connection task of the action calculation node with the center node based on the second session, and saves second address information and third address information;

[0020] The page controller receives and judges whether the second address information sent by the second session is legal, and in the case that the second address information is legal, locates the current second sample record according to the second address information, writes the current second sample record into the action sample;

[0021] The high-speed read-write module sends the third address information to the page controller through the second session, the page controller locates the third sample record according to the third address information, reads the environment state sample corresponding to the action sample according to the third sample record, and sends the environment state sample to the action calculation node.

[0022] In some embodiments, when the policy update node connects with the center node, the method comprises:

[0023] The high-speed read-write module creates a third session when connecting with the action calculation node for the first time, and in the case that the environment state sample and the action sample in the current page have been filled, obtains fourth address information corresponding to the page;

[0024] The page controller receives and judges whether the fourth address information sent by the third session is legal, and in the case that the fourth address information is legal, acquires the corresponding page according to the fourth address information, and acquires the corresponding track sample from the corresponding page;

[0025] The high-speed read-write module sends the track sample to the policy update node through the third session.

[0026] In some embodiments, the action sample is sent to the environment interaction node, and the method further comprises:

[0027] Receiving initial states of n environment instances and instance numbers corresponding to the environment instances sent by the environment interaction node, and searching for fifth address information corresponding to the environment instances through the environment numbers, wherein the initial states of the n environment instances are obtained by the environment interaction node initializing the n environment instances, and the value of n is dynamically adjusted according to the waiting time of the environment interaction node and the action calculation node, and the longer the waiting time, the greater the value of n.

[0028] Receiving the action sample, analyzing the instance number corresponding to each action sample, obtaining the updated environment state sample according to the action sample, and sending the instance number and the updated environment state sample to the center node.

[0029] In a second aspect, the embodiments of the present application provide a reinforcement learning data sampling system for storing and managing experience sets in a distributed environment, which comprises an environment interaction node, an action calculation node, a policy update node and a center node, wherein

[0030] The environment interaction node is configured to generate an environment state sample and send the environment state sample to the center node.

[0031] The action calculation node is configured to generate a corresponding action sample according to model parameters and the environment state sample and send the action sample to the center node.

[0032] The policy update node is configured to optimize a model according to a track sample and send the optimized model parameters to the action calculation node.

[0033] The center node is configured to receive and send the environment state sample to the action calculation node, receive and send the action sample to the environment interaction node, and store the environment state sample, the action sample, and the trajectory sample to the experience set. When the number of the trajectory samples stored in the experience set reaches a first threshold, the trajectory sample is sent to the policy update node.

[0034] In a third aspect, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the reinforcement learning data sampling method according to any one of the first aspect are implemented.

[0035] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program. When the program is executed by a processor, the steps of the reinforcement learning data sampling method according to any one of the first aspect are implemented.

[0036] Compared with the related art, the reinforcement learning data sampling method, system, device, and storage medium provided by the embodiment of the present application have the following advantages. The center node acquires the environment state sample generated by the environment interaction node, stores the environment state sample to the experience set, sends the environment state sample to the action calculation node, receives and stores the action sample generated by the action calculation node, sends the action sample to the environment interaction node, continues to receive and process the environment state sample updated by the environment interaction node according to the action sample, generates the trajectory sample according to the environment state sample and the action sample, sends the trajectory sample to the policy update node when the number of the trajectory samples stored in the experience set reaches a first threshold, and optimizes the model according to the trajectory sample. The optimized model parameters are sent to the action calculation node, which ensures efficient reading and writing of various node information and solves the problem of low sampling efficiency of reinforcement learning in the related art under the condition of limited computing resources.

[0037] Details of one or more embodiments of the present application are presented in the following drawings and description to make other features, objects, and advantages of the present application more apparent. BRIEF DESCRIPTION OF DRAWINGS

[0038] The accompanying drawings illustrated herein are used to provide further understanding of the present application, and constitute a part of the present application. The illustrative embodiments of the present application and their description serve to explain the present application, and do not constitute improper limitations on the present application. In the drawings:

[0039] Figure 1 is a hardware structure block diagram of a terminal of the reinforcement learning data sampling method in an embodiment of the present application;

[0040] Figure 2 is a structural diagram of a reinforcement learning data sampling system in an embodiment of the present application;

[0041] Figure 3 is a flowchart of a reinforcement learning data sampling method in an embodiment of the present application;

[0042] Figure 4 is a flowchart of a processing method of a high-speed read-write module for different nodes in the reinforcement learning data sampling method in an embodiment of the present application;

[0043] Figure 5 is a flowchart of a reinforcement learning data sampling method in another embodiment of the present application;

[0044] Figure 6 is a structural diagram of a reinforcement learning data sampling system in an embodiment of the present application. DETAILED DESCRIPTION

[0045] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application is described and explained below in conjunction with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of the present application.

[0046] Obviously, the drawings in the following description are only some examples or embodiments of the present application, and for those of ordinary skill in the art, the present application can be applied to other similar scenarios without creative efforts based on these drawings. In addition, it can be understood that although the efforts made in this development process can be complex and lengthy, for those of ordinary skill in the art related to the content disclosed in the present application, some designs, manufacturing or production changes based on the technical content disclosed in the present application are only routine technical means and should not be understood as insufficient disclosure of the present application.

[0047] In the present application, the phrase "embodiment" means that the specific features, structures or characteristics described in conjunction with the embodiment can be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily mean the same embodiment, nor is it an independent or alternative embodiment to other embodiments. Those of ordinary skill in the art explicitly and implicitly understand that the embodiments described in the present application can be combined with other embodiments without conflict.

[0048] Unless otherwise defined, technical terms and scientific terms used in the present application shall have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. Unless otherwise defined, the terms "one" and "a" or "an" used in this application do not denote a limitation of quantity, but rather denote the presence of at least one of the referenced item. The terms "including" and / or "having," as well as any variations thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, system, product or apparatus that comprises a list of steps or units is not necessarily limited to those steps or units which are recited, but can include additional steps or units, or can also use any alternatives, equivalents, or substitutes of the process, method, system, product, or apparatus to the current application. The term "plurality" means two or more.

[0049] The method embodiments provided in the present embodiment can be executed in a terminal, a computer or a similar computing device. For example, the method embodiments are executed on a terminal, Figure 1 is a hardware structure block diagram of a terminal for the reinforcement learning data sampling generation method of the present embodiment. As shown in Figure 1 , the terminal can include one or more (only one is shown in Figure 1 ) processors 102 and a memory 104 for storing data, wherein the processor 102 can include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA. The terminal can further include a transmission device 106 for communication function and an input / output device 108. As can be understood by those skilled in the art, Figure 1 , the structure shown is only schematic, and it does not limit the structure of the terminal. For example, the terminal can include more or fewer components than those shown in Figure 1 , or have a different configuration from that shown in Figure 1 .

[0050] The memory 104 can be used to store computer programs, such as software programs and modules of application software, and the computer program corresponding to the reinforcement learning data sampling method in the present embodiment. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, i.e. implements the method described above. The memory 104 can include a high-speed random access memory, and can further include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory 104 can further include a memory remotely disposed with respect to the processor 102, which can be connected to the terminal through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0051] The transmission device 106 is configured to receive or send data via a network. The network includes a wireless network provided by a communication provider of the terminal. In an example, the transmission device 106 includes a network interface controller (NIC) which is configured to connect to other network devices through a base station so as to communicate with the Internet. In an example, the transmission device 106 can be a radio frequency (RF) module which is configured to communicate with the Internet in a wireless manner.

[0052] Reinforcement learning is a kind of machine learning, and the training sample of reinforcement learning is data generated by interaction between an agent and an environment, including a state of the environment and an action generated by interaction with the environment. The agent adjusts its own behavior by means of feedback from the environment through continuous interaction with the environment, so as to maximize the cumulative return. The standard interaction process of reinforcement learning is that the agent selects an action in the current state according to a policy, the environment responds to the action, moves to a new state, and generates a reward signal, the discounted cumulative sum of the reward is a return, and the agent needs to maximize the return in the action selection process.

[0053] In the embodiment, a reinforcement learning data sampling method is provided, which is applied to model training of reinforcement learning, Figure 2 is a structural schematic diagram of a reinforcement learning data sampling system of the embodiment of the application, as Figure 2 shown, the method mainly realizes sample transmission and communication between a center node and an environment interaction node (Actor module), an action calculation node (Policy module) and a policy updating node (Learner module), wherein an environment instance in the environment interaction node collects the latest environment state sample and sends the environment state sample to the center node; the action calculation node receives the environment state sample, performs forward inference of a policy model on the environment state sample, obtains an action corresponding to each state, i.e., an action sample corresponding to the environment state sample, and sends the action sample back to the center node; and the policy updating node optimizes the model according to the trajectory sample and outputs the optimized model parameters to the action calculation node.

[0054] Figure 3 is a flowchart of the reinforcement learning data sampling method of the embodiment of the application, as Figure 3 shown, the flowchart includes the following steps:

[0055] In step S301, an environment state sample generated by an environment interaction node is acquired, and the environment state sample is stored in an experience set of a center node.

[0056] The center node receives an environment state sample from the environment interaction node and stores the environment state sample into a dataset, the environment state sample refers to a current state of an agent and current environment data, and is a series of observation values obtained from the environment and used to describe a current state of the system.

[0057] In step S302, the environment state sample is sent to the action calculation node, an action sample generated by the action calculation node according to the model parameter and the environment state sample is received, and the action sample is stored into an experience set.

[0058] The center node forwards the received environment state sample to the action calculation node, receives the action sample from the action calculation node after the action sample is generated, and stores the action sample into the experience set, the action sample refers to action data selected by the agent according to a current state in the reinforcement learning process.

[0059] In step S303, the action sample is sent to the environment interaction node, and the environment state sample updated according to the action sample by the environment interaction node is continuously received and processed; a trajectory sample is generated according to the environment state sample and the corresponding action sample in the experience set, when the number of trajectory samples stored in the experience set reaches a first threshold, the trajectory sample is sent to a policy update node, the policy update node optimizes the model according to the trajectory sample, and outputs the optimized model parameter to the action calculation node, and the first threshold is dynamically adjusted according to the model complexity, the model convergence speed and other model indicators.

[0060] The center node forwards the received action sample back to the environment interaction node, receives the environment state sample updated by the environment instance according to the action sample from the environment interaction node, and repeats the above steps S301 to S302, generates a trajectory sample according to the collected environment state sample and the corresponding action sample, and the trajectory sample is used to describe a series of state changes generated by interaction between the agent and the environment. After the above sampling process, the trajectory sample is sent to the cache of the policy update node, so that the policy node optimizes the model.

[0061] After the above sampling process, the trajectory sample is sent to the cache of the policy update node, the policy update node selects the collected sample from the sample cache area according to a certain rule, and provides the model for reinforcement learning training, the model in the policy update node optimizes the model using the trajectory sample, and updates the model parameter for the action calculation node.

[0062] Through the steps S301 to S303, the environment state sample generated by the environment interaction node is acquired by the center node, the environment state sample is stored to the experience set of the center node, the environment state sample is sent to the action calculation node, the action sample generated by the action calculation node according to the model parameter and the environment state sample is received and stored, the action sample is sent to the environment interaction node, the environment state sample updated by the environment interaction node according to the action sample is continuously received and processed, and the trajectory sample is generated according to the environment state sample and the action sample. When the number of trajectory samples stored in the experience set reaches the first threshold, the trajectory sample is sent to the policy update node, the model is optimized by the policy update node according to the trajectory sample, and the optimized model parameter is sent to the action calculation node. The calculation process in the reinforcement learning training process is divided into three parts of environment interaction, action calculation and policy update, which are deployed on three different nodes, and communicate through a center storage node without additional calculation tasks. The center node is not limited by performance, which ensures efficient reading and writing of various node information, solves the problem of low sampling efficiency of reinforcement learning in related technologies under limited computing resources, and realizes the beneficial effect of improving the sampling efficiency of reinforcement learning under limited computing resources.

[0063] In some embodiments, the center node includes a high-speed read-write module, which is used to create a session between the center node and the target node, and perform read-write operations on sample data based on the session. This can improve data transmission speed and thus improve the efficiency of the entire system. The target node includes an environment interaction node, an action calculation node, or a policy update node, and the sample data includes an environment state sample, an action sample, or a trajectory sample.

[0064] As shown in Figure 3 The high-speed read-write module in the center node undertakes a large number of read-write operations, so the high-speed read-write module uses gRPC (remote procedure call communication framework) as the network communication framework. The high-speed read-write module processes different nodes connected thereto differently.

[0065] In some embodiments, the high-speed read-write module page-encapsulates the storage space of the experience set, and each page of storage space corresponds to different address information, and the high-speed read-write module performs parallel reading and writing on different pages of storage space.

[0066] The high-speed read-write module encapsulates the storage space of the experience set to improve the reading and writing efficiency of the nodes, and designs a paging mechanism. A page is a logical address space division, which is a logical space sequentially divided from a logical address space and numbered consecutively. Page encapsulation helps improve memory usage efficiency and reduce fragmentation. In this way, the high-speed read-write module ensures the parallel reading and writing efficiency of the center storage when a large number of distributed nodes are connected to the center node simultaneously.

[0067] In some embodiments, the high-speed read-write module further comprises a page controller, which is important in the whole system because it helps to improve the data retrieval speed and reduce the delay. The page controller is used to determine whether the address information is available, and to find the page storage information corresponding to the address information according to the address information.

[0068] The page controller is used to determine whether the page ID (i.e. the address information of the page) is available or legal, to obtain the corresponding page according to the page ID, and to locate the current sample record from the corresponding page.

[0069] In some embodiments, when the environment interaction node is connected with the center node:

[0070] The high-speed read-write module creates a first session when it is first connected with the environment interaction node, processes the connection task of the environment interaction node and the center node based on the first session, and is used to save the first address information; the page controller receives and judges whether the first address information sent by the first session is available, and in the case that the first address information is available, locates the current first sample record according to the first address information, writes the current first sample record into the environment state sample, updates the first address information, and sends the updated first address information to the environment interaction node.

[0071] In some embodiments, the high-speed read-write module comprises a page controller, which is used to determine whether the address information is available, and to find the page storage information corresponding to the address information according to the address information. Figure 4 is the flowchart of the processing method of the high-speed read-write module for different nodes in the reinforcement learning data sampling method of the embodiments of the present application, as shown in Figure 4As shown, when the high-speed read-write module processes the read-write operation of the environment interaction node (Actor module), the environment interaction node will establish a connection while sending the environment state sample to the center node for the first time, and the high-speed read-write module will create a first session to save the page ID (first address information) currently being operated. After that, the connection between the high-speed read-write module and the environment interaction node will use the same session for read-write operation to avoid frequent connection operations. After the session is created successfully, the current first session sends the first address information currently needing to be operated to the page controller through the cache interface. The page controller checks the received first address information. If the page corresponding to the first address information is available, it directly performs subsequent read-write operation, otherwise it needs to obtain a new page under the condition of locking and return the corresponding address. In the case of judging that the current page is available, the page controller obtains the corresponding page according to the first address information and defines the callback function required for the return action. In the corresponding page, the current sample record can be located, and the environment state sample is written in the memory corresponding to the current sample record to complete the write operation of the environment interaction node. When the action corresponding to the environment state sample is ready, the action computing node calls the callback function defined by the page controller to return the corresponding action sample and new address information to the cache interface of the session. The current session saves the new address information and sends it back to the environment interaction node to complete the read operation of the environment interaction node.

[0072] In some embodiments, when the action computing node connects with the center node:

[0073] The high-speed read-write module creates a second session when connecting with the action computing node for the first time, processes the connection task of the action computing node and the center node based on the second session, and saves the second address information and the third address information;

[0074] The page controller receives and judges whether the second address information sent by the second session is legal. In the case that the second address information is legal, the current second sample record is located according to the second address information, and the current second sample record is written into the action sample;

[0075] The high-speed read-write module sends the third address information to the page controller through the second session. The page controller locates the third sample record according to the third address information, reads the environment state sample corresponding to the action sample according to the third sample record, and sends the environment state sample to the action computing node.

[0076] In some embodiments, the high-speed read-write module includes: Figure 4As shown, the action computing node (Policy module) sends the action corresponding to the previously received environment state to the central node. The initial transmission is empty, a connection is established, and the high-speed read / write module creates a second session to store the second address information corresponding to the current action sample, i.e., the corresponding page ID. Subsequent connections will use the second session for read / write operations to avoid frequent connections. The second session sends the second address information corresponding to the current action sample to the page controller through a cache interface. The page controller checks the validity of the second address information. If valid, it retrieves the corresponding page, locates the current sample record within the page, and writes the action sample into the corresponding memory, completing the write operation of the action computing node. The callback function of the current page is called to notify the environment interaction node that the action sample is ready. Simultaneously, the second session retrieves the third address information of the page currently waiting for the action. The page controller retrieves the corresponding page based on the third address information, locates the target sample record within the page, reads the environment state sample from the memory corresponding to the target sample record, and sends the corresponding environment state sample back to the action computing node through the second session, completing the read operation of the action computing node.

[0077] In some embodiments, when the policy update node connects to the central node:

[0078] When the high-speed read / write module first connects to the action computing node, it creates a third session and obtains the fourth address information corresponding to the page when the environment state sample and action sample in the current page are already filled.

[0079] The page controller receives and determines whether the fourth address information sent by the third session is valid. If the fourth address information is valid, it obtains the corresponding page based on the fourth address information and obtains the trajectory sample from the corresponding page.

[0080] The high-speed read / write module sends trajectory samples to the policy update node through a third session.

[0081] Among them, such as Figure 4 As shown, the policy update node sends a command to the central node to read trajectory samples. A connection is established upon the first transmission, and the high-speed read / write module creates a third session. Subsequent connections will use this third session for read / write operations to avoid frequent connections. The page currently filled with environment state and action samples is selected. The third session sends the fourth address information (page ID) corresponding to the current action sample to the page controller via the cache interface. The page controller checks the validity of the fourth address information. If the fourth address information is valid, the corresponding page is retrieved based on the fourth address information, the trajectory sample is read from the page, and the trajectory sample is sent back to the policy update module via the third session, completing the read operation of the policy update module.

[0082] In some embodiments, the action sample is sent to the environment interaction node, and the step of continuously receiving and processing the environment state sample updated by the environment interaction node according to the action sample comprises:

[0083] The initial state of the n environment instances sent by the environment interaction node and the instance number corresponding to the environment instance are received, and the fifth address information corresponding to the environment instance is found through the environment number, wherein the initial state of the n environment instances is obtained by initializing the n environment instances by the environment interaction node, and the value of n is dynamically adjusted according to the waiting time of the environment interaction node and the action calculation node, and the longer the waiting time, the larger the value of n;

[0084] The action sample is received, the instance number corresponding to each action sample is parsed, the updated environment state sample is obtained according to the action sample, and the instance number and the updated environment state sample are sent to the center node.

[0085] In order to achieve the most ideal utilization efficiency of computing resources, it is necessary to dynamically adjust the sampling rate and the training rate for different training conditions. Therefore, the system of the embodiment of the application adopts a multi-buffer technology to dynamically adjust the number of environment instances in the environment interaction node, so that the sampling rate can be matched with the training rate. Assuming that the number of initial environment instances is n, for a certain environment interaction node, as shown in the figure, the overall process of its operation is: Figure 5

[0086] Step S501, the environment interaction node and the center node establish a connection and prepare to transfer sample information;

[0087] Step S502, initialize n environment instances, and send the initial state of each instance and the instance number of the corresponding instance to the center node in turn, wherein the value of n can be dynamically adjusted according to the waiting time of the environment interaction node and the action calculation node, and when the waiting time of the environment interaction node is large, the value of n is appropriately increased, and vice versa, to reduce the communication overhead;

[0088] Step S503, real-time monitoring of the connection established with the center node, whenever an action sample is received, the instance number of the environment instance corresponding to the action sample is parsed, the received action is used to perform environment deduction on the corresponding environment, the new environment state and the reward signal are obtained, and the environment state, the reward signal and the instance number of the corresponding environment instance are sent back to the center node.

[0089] ​The embodiment of the application uses a multi-cache technology sampling method, uses the time for policy inference of the action calculation node to perform environment deduction operation, makes up for the waiting time of the action calculation node, and improves the sampling efficiency. In actual operation, the value of n can be dynamically adjusted according to the waiting time of the environment interaction node and the action calculation node, the efficiency of dynamic balance sampling and training is realized, and the use efficiency of the computing resource is optimized.

[0090] The embodiment also provides a reinforcement learning data sampling system for storing and managing experience sets in a distributed environment, so as to further improve the performance of the data sampling system in the distributed environment. Figure 6 The reinforcement learning data sampling system is shown in FIG. 1, which comprises an environment interaction node 61, an action calculation node 62, a policy update node 63 and a center node 64. Figure 5 The environment interaction node 61 is configured to generate an environment state sample and send the environment state sample to the center node 64.

[0091] The action calculation node 62 is configured to generate a corresponding action sample according to the model parameters and the environment state sample, and send the action sample to the center node 64.

[0092] The policy update node 63 is configured to optimize the model according to the trajectory sample, and send the optimized model parameters to the action calculation node 62.

[0093] The center node 64 is configured to receive and send the environment state sample to the action calculation node 62, receive and send the action sample to the environment interaction node 61, and store the environment state sample, the action sample and the trajectory sample to the experience set. When the number of trajectory samples stored in the experience set reaches a first threshold, the trajectory sample is sent to the policy update node 63.

[0094] The embodiment of the application also provides a computer device comprising a memory and a processor. The memory stores a computer program. The processor implements the steps of the reinforcement learning data sampling method provided in the above embodiments when executing the computer program.

[0095] The embodiment of the application also provides a computer readable storage medium, which stores a computer program. The computer program is executed by the processor to implement the steps of the reinforcement learning data sampling method provided in the above embodiments.

[0096]

[0097] ​Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, storage, databases, or other media in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in many forms such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (SyncLink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0098] The technical features of the above-mentioned embodiments can be combined in any way. In order to make the description simple, all possible combinations of the technical features in the above-mentioned embodiments are not described, but as long as the combination of the technical features does not exist, it should be considered as the scope of the present application.

[0099] The above-mentioned embodiments only express several embodiments of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of the present application. Therefore, the scope of the patent protection of the present application should be subject to the appended claims.

Claims

1. A method for reinforcement learning data sampling, applied to model training of reinforcement learning, characterized in that, The method comprises: obtaining an environment state sample generated by an environment interaction node, and storing the environment state sample to an experience set of a central node; the central node comprises a high-speed read-write module, the high-speed read-write module is used for creating a session between the central node and a target node, and performing read-write operations on sample data based on the session, wherein the target node comprises the environment interaction node, an action calculation node or a policy update node, and the sample data comprises the environment state sample, an action sample or a trajectory sample; the high-speed read-write module performs page encapsulation on a storage space of the experience set, and each page of the storage space corresponds to different address information; and the high-speed read-write module performs parallel read-write on the storage space of different pages; sending the environment state sample to the action calculation node, receiving the action sample generated by the action calculation node based on model parameters and the environment state sample, and storing the action sample to the experience set; sending the action sample to the environment interaction node, continuing to receive and process the environment state sample updated by the environment interaction node based on the action sample; generating the trajectory sample based on the environment state sample and the corresponding action sample in the experience set; when the number of the trajectory samples stored in the experience set reaches a first threshold, sending the trajectory sample to the policy update node, the policy update node optimizing a model based on the trajectory sample, and outputting the optimized model parameters to the action calculation node, wherein the first threshold is dynamically adjusted based on model complexity, model convergence speed and other model indicators.

2. The reinforcement learning data sampling method of claim 1, wherein, The high-speed read-write module further comprises a page controller, the page controller is used for judging whether the address information is available, and searching for page storage information corresponding to the address information based on the address information.

3. The reinforcement learning data sampling method of claim 2, wherein, When the environment interaction node is connected to the central node, the method comprises: the high-speed read-write module creates a first session when first connected to the environment interaction node, processes connection tasks of the environment interaction node and the central node based on the first session, and is used for saving first address information; the page controller receives and judges whether the first address information sent by the first session is available, in the case that the first address information is available, locates a current first sample record based on the first address information, writes the current first sample record to the environment state sample, updates the first address information, and sends the updated first address information to the environment interaction node.

4. The reinforcement learning data sampling method of claim 2, wherein, When the action calculation node is connected to the central node, the method comprises: the high-speed read-write module creates a second session when first connected to the action calculation node, processes connection tasks of the action calculation node and the central node based on the second session, and saves second address information and third address information; The page controller receives and judges whether the second address information sent by the second session is legal, and in the case that the second address information is legal, locates the current second sample record according to the second address information, and writes the current second sample record into the action sample; The high-speed read-write module sends the third address information to the page controller through the second session, and the page controller locates the third sample record according to the third address information, reads the environment state sample corresponding to the action sample according to the third sample record, and sends the environment state sample to the action calculation node.

5. The reinforcement learning data sampling method of claim 2, wherein, When the policy updating node is connected with the center node, the method comprises: The high-speed read-write module creates a third session when connecting with the action calculation node for the first time, and acquires fourth address information corresponding to the current page in the case that the environment state sample and the action sample in the current page have been filled; The page controller receives and judges whether the fourth address information sent by the third session is legal, and in the case that the fourth address information is legal, acquires the corresponding current page according to the fourth address information, and acquires the corresponding trajectory sample from the corresponding current page; The high-speed read-write module sends the trajectory sample to the policy updating node through the third session.

6. The reinforcement learning data sampling method of claim 1, wherein, Sending the action sample to the environment interaction node, and continuing to receive and process the environment state sample updated by the environment interaction node according to the action sample comprises: Receiving the initial state of n environment instances and the instance number corresponding to the environment instance sent by the environment interaction node, and searching for fifth address information corresponding to the environment instance through the instance number corresponding to the environment instance, wherein the initial state of the n environment instances is obtained by initializing the n environment instances by the environment interaction node, and the value of n is dynamically adjusted according to the waiting time of the environment interaction node and the action calculation node, and the longer the waiting time, the larger the value of n; Receiving the action sample, analyzing the instance number corresponding to each action sample, obtaining the updated environment state sample according to the action sample, and sending the instance number and the updated environment state sample to the center node.

7. A reinforcement learning data sampling system for storing and managing a set of experiences in a distributed environment, the system comprising: Comprise: An environment interaction node, an action calculation node, a policy updating node and a center node; wherein The environment interaction node is configured to generate an environment state sample and send the environment state sample to the center node; The action calculation node is configured to generate a corresponding action sample according to a model parameter and the environment state sample, and send the action sample to the center node; The policy updating node is configured to optimize a model according to a trajectory sample, and send the optimized model parameter to the action calculation node; and The center node is configured to receive the environment state sample, the action sample and the trajectory sample, and send the environment state sample and the action sample to the action calculation node, and send the trajectory sample to the policy updating node. The center node is configured to receive and send the environment state samples to the action calculation node, receive and send the action samples to the environment interaction node, and store the environment state samples, the action samples, and the trajectory samples to the experience set. When the number of the trajectory samples stored in the experience set reaches a first threshold, the trajectory samples are sent to the policy update node. The center node comprises a high-speed read-write module configured to create a session between the center node and a target node, and perform read-write operations on sample data based on the session, wherein the target node comprises the environment interaction node, the action calculation node, or the policy update node, and the sample data comprises the environment state samples, the action samples, or the trajectory samples. The high-speed read-write module performs paging encapsulation on a storage space of the experience set, and each page of the storage space corresponds to different address information. The high-speed read-write module performs parallel read-write on the storage space of different pages.

8. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the reinforcement learning data sampling method according to any one of claims 1 to 6.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the steps of the reinforcement learning data sampling method according to any one of claims 1 to 6.

Images