A reinforcement learning training method and device based on a ray framework and a medium

Abstract

The application discloses a reinforcement learning training method and device based on a ray framework and a medium, and relates to the technical field of data processing. The method comprises the following steps: collecting sample data of interaction between an agent and an environment in parallel based on a first preset number of data collection workers; obtaining different batches of sample data in parallel based on a second preset number of sub-training nodes, wherein the sub-training nodes and the data collection workers are modified through a decorator of the ray framework; performing gradient calculation on the sample data obtained by each sub-training node in parallel to obtain local gradients corresponding to the agent and different batches of sample data; and aggregating all the local gradients based on a parameter server to obtain a global gradient of the agent, updating global parameters of the agent according to the global gradient, and obtaining a trained agent. The application improves the training efficiency of reinforcement learning by collecting sample data in parallel and performing agent gradient calculation in parallel.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a reinforcement learning training method, device and medium based on the ray framework. Background Technology

[0002] Reinforcement learning, as a machine learning method that learns optimal strategies through interaction between an agent and its environment, has enormous potential for application in complex scenarios such as robot control, autonomous driving, and game AI. However, as scenario complexity increases, traditional reinforcement learning training faces numerous challenges.

[0003] In serial training, complex scenarios often involve high-dimensional state and action spaces. Traditional serial training updates only one model parameter per time step, leading to an exponential increase in computational complexity when handling massive amounts of data, such as control tasks in high-resolution image states, where Q-tables or neural networks update slowly. Simultaneously, serial training samples are underutilized; the agent selects only one action at a time, making it difficult to explore multiple strategies in parallel. For example, maze pathfinding requires numerous trials to find the optimal path. Moreover, in large-scale interactive tasks, serial training is extremely time-consuming; for instance, robot grasping training can take days or even weeks, limiting practical applications. Furthermore, in single-machine, multi-GPU serial training scenarios, traditional frameworks suffer from significant parameter synchronization problems. Each computing node independently calculates gradients and then serially synchronizes them to the master node, resulting in a large number of idle GPUs and wasted computing resources. Frequent parameter push and pull communication incurs high overhead, consuming network bandwidth. As the number of GPUs increases, communication time becomes a bottleneck, affecting training throughput and further reducing training efficiency.

[0004] In summary, how to provide a parallel reinforcement learning training method based on the Ray framework to improve the training efficiency of reinforcement learning in existing technologies is an urgent problem to be solved. Summary of the Invention

[0005] This application provides a reinforcement learning training method, device, and medium based on the Ray framework. The main purpose is to provide a parallel reinforcement learning training method based on the Ray framework to improve the training efficiency of reinforcement learning in the prior art.

[0006] In a first aspect, embodiments of this application provide a reinforcement learning training method based on the Ray framework. The method includes: collecting sample data of the interaction between the agent and the environment in parallel using a first preset number of data acquisition workers, wherein the first preset number of data acquisition workers are all decorated using decorators of the Ray framework.

[0007] The sample data of different batches are acquired in parallel based on a second preset number of sub-training nodes. During the training process, there are preset constraints between the second preset number and the first preset number. All sub-training nodes of the second preset number are decorated by the decorator.

[0008] Based on the second preset number of sub-training nodes, gradient calculations are performed in parallel on the sample data acquired by each node to obtain the local gradients of the agent and the sample data from different batches.

[0009] The parameter server aggregates all the local gradients to obtain the global gradient of the agent, and updates the global parameters of the agent according to the global gradient to obtain the trained agent.

[0010] In one implementation of this application, after acquiring different batches of sample data in parallel based on a second preset number of sub-training nodes, the method further includes:

[0011] Different batch IDs are set for different batches of the sample data;

[0012] Bind different batch IDs to different cache threads;

[0013] When it is determined that the number of sample data in any batch of the sample data in different batches meets the preset number threshold, the cache thread bound to the batch ID caches the sample data of that batch to the replay cache area, and the different cache threads execute the caching task of the sample data in parallel.

[0014] In one implementation of this application, the step of acquiring different batches of sample data in parallel based on a second preset number of sub-training nodes includes:

[0015] The sample data from different batches in the replay buffer are uniformly matched to the second preset number of sub-training nodes;

[0016] The second preset number of sub-training nodes retrieve their respective matched sample data from the replay buffer in parallel.

[0017] In one implementation of this application, the parallel collection of sample data on the interaction between the agent and the environment by a first preset number of data acquisition workers includes:

[0018] Multiple independent environment instances are generated based on the Gym environment interface;

[0019] The first preset number of data acquisition workers load the plurality of independent environment instances respectively, so that each of the first preset number of data acquisition workers executes the interaction strategy between the agent and the loaded environment instance to obtain the sample data corresponding to each of them. Each of the first preset number of data acquisition workers loads one of the plurality of independent environment instances, and the environment instances loaded by each of the data acquisition workers are different.

[0020] In one implementation of this application, the step of performing gradient calculations in parallel on the sample data acquired by each of the sub-training nodes based on the second preset number of sub-training nodes to obtain the local gradients of the agent corresponding to different batches of sample data includes:

[0021] Load copies of the agent's policy network and value network into the second preset number of sub-training nodes;

[0022] The sample data corresponding to each sub-training node is input into the policy network copy and the value network copy in the corresponding sub-training node to calculate the policy gradient loss and value loss corresponding to each sub-training node.

[0023] Based on the policy gradient loss and the value loss corresponding to each sub-training node, the total loss corresponding to each sub-training node is calculated.

[0024] Based on the total loss corresponding to each sub-training node, the local gradients of the agent and the sample data of different batches are calculated.

[0025] In one implementation of this application, the global gradient of the agent is obtained by aggregating all the local gradients based on a parameter server, including:

[0026] Based on the parameters, the server receives the local gradients corresponding to the agent and different batches of sample data;

[0027] The global gradient is obtained by performing a weighted average of all received local gradients.

[0028] In one implementation of this application, the method includes: if the parameter server uses a synchronous aggregation method during the aggregation of the local gradients, then based on a first preset number of data acquisition workers, sample data of the agent's interaction with the environment are collected in parallel; based on a second preset number of sub-training nodes, different batches of the sample data are acquired in parallel; and based on the second preset number of sub-training nodes, gradient calculations are performed in parallel on the sample data acquired by each of them to obtain the local gradients corresponding to the agent and the sample data of different batches. These three steps are executed in parallel. Based on the parameter server, all the local gradients are aggregated to obtain the global gradient of the agent. This step is executed independently of the three steps.

[0029] If the parameter server adopts an asynchronous aggregation method, the three steps are executed in parallel with the current step. The synchronous aggregation method is to perform local gradient aggregation when it is determined that the parameter server has received all local gradients. The asynchronous aggregation method is to perform local gradient aggregation when it is determined that the parameter server has received any number of local gradients.

[0030] In one implementation of this application, after obtaining the trained agent, the method further includes:

[0031] The model demonstration class showcases the effectiveness of the trained agent.

[0032] The effectiveness of the application is evaluated based on the model evaluation class to obtain the performance evaluation index of the agent. Both the model evaluation class and the model display class are decorated with the decorator and then executed in parallel.

[0033] Secondly, embodiments of this application also provide a reinforcement learning training device based on the Ray framework. The device includes: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores instructions executable by the at least one processor, which, when executed, enable the at least one processor to: acquire sample data of agent-environment interaction in parallel using a first preset number of data acquisition workers, each of which is decorated using a decorator of the Ray framework; acquire different batches of the sample data in parallel using a second preset number of sub-training nodes, where the second preset number and the first preset number are subject to preset constraints during training, and each of the second preset number of sub-training nodes is decorated using the decorator; perform gradient calculations in parallel on the sample data acquired by each of the second preset number of sub-training nodes to obtain local gradients corresponding to the agent and different batches of the sample data; aggregate all the local gradients based on a parameter server to obtain the global gradient of the agent, and update the global parameters of the agent according to the global gradient to obtain the trained agent.

[0034] Thirdly, embodiments of this application also provide a non-volatile computer storage medium for reinforcement learning training based on the Ray framework, storing computer-executable instructions. These instructions are configured to: acquire sample data of agent-environment interaction in parallel using a first preset number of data acquisition workers, each of which is decorated using a decorator of the Ray framework; acquire different batches of the sample data in parallel using a second preset number of sub-training nodes, where the second preset number and the first preset number are subject to preset constraints during training, and each of the second preset number of sub-training nodes is decorated using the decorator; perform gradient calculations in parallel on the sample data acquired by each of the second preset number of sub-training nodes to obtain local gradients corresponding to the agent and different batches of the sample data; aggregate all the local gradients using a parameter server to obtain the global gradient of the agent, and update the global parameters of the agent based on the global gradient to obtain the trained agent.

[0035] This application provides a reinforcement learning training method, device, and medium based on the Ray framework, which has the following beneficial effects: By setting a first preset number of data acquisition workers modified by decorators, the data acquisition workers are distributed to different CPU cores for parallel execution, thereby realizing the parallel acquisition of sample data of agent-environment interaction and improving the sample data acquisition efficiency. Similarly, by setting a second preset number of sub-training nodes modified by decorators, the parallel acquisition of sample data in different batches and the parallel calculation of corresponding local gradients based on the sample data in different batches are realized, avoiding the scenario of gradient calculation for large-scale sample data and reducing the complexity of gradient calculation. In addition, by acquiring the sample data in parallel for gradient calculation, the matching and unification of sample data acquisition efficiency and GPU gradient calculation efficiency are achieved, improving the utilization of GPU resources and further improving the training efficiency of reinforcement learning. Attached Figure Description

[0036] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0037] Figure 1 A flowchart of a reinforcement learning training method based on the Ray framework provided in this application embodiment;

[0038] Figure 2 A schematic diagram of a ray-based reinforcement learning training framework provided in an embodiment of this application;

[0039] Figure 3 This is a schematic diagram of the internal structure of a reinforcement learning training device based on the ray framework, provided as an embodiment of this application. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0041] To facilitate understanding of the following embodiments of this application, some conceptual explanations and definitions are provided, including: Reinforcement learning mainly consists of an agent, an environment, a state, an action, and a reward. After the agent performs an action, the environment transitions to a new state and provides a reward signal (positive or negative) for this new state. Subsequently, the agent executes a new action according to a certain strategy based on the new state and the reward feedback from the environment. The above process describes the interaction between the agent and the environment through state, action, and reward. Through reinforcement learning, the agent can know what action it should take in what state to obtain the maximum reward. Since the interaction between the agent and the environment is similar to the interaction between humans and the environment, reinforcement learning can be considered a general learning framework that can be used to solve the problems of general artificial intelligence. Therefore, reinforcement learning is also called a machine learning method for general artificial intelligence. In this system, the agent represents the ontology of reinforcement learning, acting as a learner or decision-maker; the environment represents everything outside the agent, primarily consisting of a set of states; the state represents the data of the environment, and the state set is all possible states in the environment; the action represents the actions the agent can perform, and the action set is all the actions the agent can perform; the reward represents the positive / negative feedback signal the agent receives after performing an action, and the reward set is all the feedback information the agent can obtain; the policy represents the learning of the mapping from environment states to actions, and this mapping relationship is called the policy. In simpler terms, the thought process by which the agent chooses actions is called the policy; the goal represents finding the optimal policy in a continuous time series, and the optimal policy usually refers to maximizing the long-term cumulative reward.

[0042] This application provides a reinforcement learning training method, device, and medium based on the Ray framework to solve the following technical problem: how to provide a parallel reinforcement learning training method based on the Ray framework to improve the training efficiency of reinforcement learning in the prior art.

[0043] The technical solutions proposed in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0044] Figure 1 This document provides a flowchart of a reinforcement learning training method based on the Ray framework, as illustrated in an embodiment of this application. Figure 1 As shown in the figure, the reinforcement learning training method based on the Ray framework provided in this application embodiment specifically includes the following steps:

[0045] Step 101: Based on a first preset number of data acquisition workers, sample data of the interaction between the agent and the environment are collected in parallel. The first preset number of data acquisition workers are all decorated using the decorator of the ray frame.

[0046] As a refinement of step 101 above, the selection of the first preset number can be determined based on the number of CPU cores executing the data acquisition task. For example, if a CPU core has 4 cores and each core executes two data acquisition workers, then the first preset number can be set to 8. It should be understood that the foregoing explanation is only for ease of understanding and does not limit the first preset number in this embodiment to necessarily being set to 8. In addition, in order to achieve parallel acquisition of the sample data, each acquisition worker is decorated with the decorator to achieve parallel operation of each acquisition worker. Specifically, the decorator is the ray.remote decorator from the ray open-source library.

[0047] To facilitate understanding of the above embodiments, this application provides a brief description of the Ray open-source library and the `ray.remote` decorator, including: Ray is an open-source library for parallel and distributed Python. Ray's advantages are as follows: it provides an efficient data sharing mechanism, enabling cross-node data access through object storage; it supports flexible allocation of computing resources; and it provides comprehensive exception handling and retry mechanisms. Ray's core foundation is the `remote` function, which gives developers the ability to dynamically customize computational dependencies (task DAGs). To run a function in the background or execute it in parallel, simply add the `ray.remote` decorator. When calling it, instead of directly calling the function name, the `remote` method is called, which immediately returns a function object, informing the developer that the task has been submitted, but the result is not yet available. The developer needs to use `ray.get` to obtain the final result of this function object. Furthermore, `ray.remote` can also decorate a class, where each instance of the class is an Actor. Each Actor has its own independent memory space to store state, such as counters and caches. Calling an Actor's method is essentially sending it a message. The Actor processes these messages sequentially, updates its state, and returns the result. This approach naturally avoids complex locking mechanisms and is well-suited for handling stateful concurrency issues.

[0048] Step 102: Based on the second preset number of sub-training nodes, different batches of sample data are acquired in parallel. During the training process, the second preset number and the first preset number are subject to preset constraints. The second preset number of sub-training nodes are all decorated by the decorator.

[0049] As a refinement of step 102 above, to ensure the efficiency of the sub-training nodes in reading the sample data, each sub-training node is decorated with the ray.remote decorator to achieve parallel operation among the sub-training nodes, thereby realizing parallel acquisition of the sample data. Simultaneously, to ensure that the efficiency of the data acquisition worker in acquiring the sample data during training matches the efficiency of the sub-training nodes in acquiring the sample data, a preset constraint is set between the first preset quantity and the second preset quantity. For example, a preset ratio relationship is set between the first preset quantity and the second preset quantity. During training, based on a load balancing mechanism, regardless of whether the first preset quantity or the second preset quantity is used, the other will change accordingly to maintain the preset ratio relationship. It should be understood that the preset ratio relationship used as a constraint in this embodiment is merely exemplary. In addition, the constraint can also be limited to a preset functional relationship. This embodiment does not limit the specific implementation details of the constraint.

[0050] Step 103: Based on the second preset number of sub-training nodes, perform gradient calculations in parallel on the sample data acquired by each node to obtain the local gradients of the agent and the sample data of different batches.

[0051] As a refinement of step 103 above, in order to prevent the complexity of gradient calculation from increasing exponentially due to too much sample data, the local gradient of the agent training is calculated by running the sub-training nodes in parallel, which effectively reduces the data scale and the complexity of gradient calculation.

[0052] Step 104: Aggregate all the local gradients based on the parameter server to obtain the global gradient of the agent, and update the global parameters of the agent according to the global gradient to obtain the trained agent.

[0053] As a refinement of the above embodiments, one or more parameter servers can be set. If multiple parameter servers are set, each parameter server can be decorated with the `ray.remote` decorator to achieve parallel operation of multiple parameter servers. Since the local gradients calculated by multiple sub-training nodes may be biased due to differences in the distribution of local sample data, parameter servers are used to aggregate all the local gradients to neutralize noise in all local gradients, smooth fluctuations in all local gradients, and then calculate the global gradient. The global gradient is used to update the global parameters of the agent, making the global parameters closer to the global optimum and avoiding getting trapped in local optima. Ultimately, this improves training efficiency while ensuring the performance of the trained agent.

[0054] This application provides a reinforcement learning training method based on the Ray framework, comprising: distributing each data acquisition worker to different CPU cores for parallel execution by setting a first preset number of data acquisition workers modified by decorators, thereby realizing the parallel acquisition of sample data of agent-environment interaction and improving the sample data acquisition efficiency; similarly, setting a second preset number of sub-training nodes modified by decorators to achieve parallel acquisition of sample data in different batches and parallel calculation of corresponding local gradients based on sample data in different batches, avoiding the scenario of gradient calculation on large-scale sample data and reducing the complexity of gradient calculation; furthermore, by acquiring the sample data in parallel for gradient calculation, the efficiency of sample data acquisition and GPU gradient calculation is matched and unified, improving the utilization of GPU resources and further improving the training efficiency of reinforcement learning.

[0055] As a refinement of the embodiments of this application, after executing step 102, in which the sample data of different batches are acquired in parallel by the sub-training nodes based on the second preset number, the method may also adopt, but is not limited to, the following implementation methods, for example: setting different batch IDs for the sample data of different batches; binding different batch IDs to different cache threads; when it is determined that the number of sample data in any batch of the sample data in different batches meets the preset number threshold, the cache thread bound to the batch ID caches the sample data of that batch to the replay buffer, and the different cache threads execute the caching task of the sample data in parallel.

[0056] To facilitate understanding of the above embodiments, this embodiment provides an exemplary description, including: If 10,000 sample data need to be processed, the batch size is set to 100 (i.e., each batch consists of 100 data items), the preset quantity threshold is 100 (i.e., caching is triggered when the batch data is full), and independent caching threads are allocated to different batches. The sample data is divided into 100 batches according to the data order, and each batch is assigned a unique batch ID: batch_001, batch_002...batch_100, with each ID corresponding to a range of 100 samples (e.g., batch_001 corresponds to samples 1-100, batch_002 corresponds to samples 101-200, etc.). An independent caching thread is bound to each batch ID: thread_001 is bound to batch_001, thread_002 is bound to batch_002, and so on, creating a total of 100 caching threads, which run independently of each other. Once the sub-training node has processed 100 data entries in batch_001 (meeting the threshold of 100), thread_001 immediately starts and caches this batch of data in the replay buffer. Simultaneously, if the 100 data entries in batch_005 have also been processed, thread_005 does not need to wait for other threads and starts its caching task synchronously. Even if some batches (such as batch_010) have not yet been processed (data volume less than 100), the batches that have met the conditions can still be cached in parallel by their respective bound threads without affecting each other. It should be understood that the foregoing description is merely illustrative and does not constitute a limitation of this disclosure.

[0057] As a refinement of the above embodiments, when performing step 102, which involves parallel acquisition of different batches of sample data by a second preset number of sub-training nodes, the following implementation methods may also be adopted, for example: uniformly matching the sample data of different batches in the replay buffer to the second preset number of sub-training nodes; and parallel acquisition of the matched sample data from the replay buffer by the second preset number of sub-training nodes.

[0058] To facilitate understanding of the above embodiments, this embodiment provides an exemplary description, including: setting a second preset number of sub-training nodes to 4 (numbered Node1~Node4), replaying 12 batches of sample data (batch IDs are batch_001~batch_012) already cached in the buffer, with each batch containing the same number of samples. The 12 batches are evenly distributed among the 4 nodes according to a "round-robin" allocation principle: Node1 is matched to batch_001, batch_005, and batch_009; Node2 is matched to batch_002, batch_006, and batch_010; Node3 is matched to batch_003, batch_007, and batch_011; and Node4 is matched to batch_004, batch_008, and batch_012. This allocation method ensures that each node is responsible for 3 batches, achieving a uniform distribution of data volume. If the total number of batches is not divisible by the number of nodes (e.g., 13 batches), the first few nodes will be allocated one more batch (e.g., Node1 will be allocated an extra batch_013) to maintain overall data balance. The four nodes simultaneously retrieve their respective batches from the cache: Node1 reads batch_001, batch_005, and batch_009 through an independent data channel; Node2 reads batch_002, batch_006, and batch_010 in parallel, without interfering with Node1's operations; Node3 and Node4 also read their assigned batches in parallel in the same way. Since each node only processes its assigned batch, and the reading process is independent, resource contention is avoided. Even if a node's retrieval speed is slow due to network or hardware issues (e.g., Node3 takes a long time to read batch_007), other nodes can still complete their tasks normally, ensuring overall data acquisition efficiency. This approach ensures load balance across the training nodes and fully utilizes computing resources through parallel operations, improving the overall throughput of data acquisition. It should be understood that the foregoing description is merely illustrative and does not constitute a limitation of this disclosure.

[0059] As a refinement of the above embodiments, when performing step 101, which involves parallel collection of sample data of agent-environment interaction by a first preset number of data acquisition workers, the following implementation methods can also be adopted, but are not limited to: generating multiple independent environment instances based on the Gym environment interface; loading the multiple independent environment instances by the first preset number of data acquisition workers respectively, so that each of the first preset number of data acquisition workers executes the interaction strategy between the agent and the loaded environment instance to obtain the sample data corresponding to it, wherein any one of the first preset number of data acquisition workers loads one of the multiple independent environment instances, and the environment instances loaded by each of the data acquisition workers are different.

[0060] To facilitate understanding of the above embodiments, this embodiment provides an exemplary description, including: If a reinforcement learning system needs to train an agent to play the Atari game "Breakout", a first preset quantity is set to 4 (i.e., 4 data acquisition workers) to collect interaction sample data in parallel. Four completely independent "Breakout" environment instances are created using the `gym.make()` method of the `Gym` interface, labeled as `env_001`, `env_002`, `env_003`, and `env_004`, respectively. These instances have their own independent memory space, game state (such as ball position, block distribution, etc.), and random seed. Even with the same input action, different interaction processes will occur due to differences in the initial state. Four data acquisition workers (numbered Worker1~Worker4) are started, each worker loading a unique environment instance: Worker1 loads `env_001`; Worker2 loads `env_002`; Worker3 loads `env_003`; Worker4 loads `env_004`; ensuring that the environment instance corresponding to each worker is unique to avoid state interference. Four workers simultaneously drive the agent to interact with its respective loaded environment instance: Worker1 controls the agent to perform actions (such as moving a paddle left or right) in env_001, acquiring sample data containing "the current game screen, the performed action, the score reward, and the next screen"; Worker2 independently executes the same strategy in env_002, generating a completely different sample sequence from Worker1 due to the different initial state of the environment; Worker3 and Worker4 interact in parallel in the same way, each generating independent samples for its corresponding environment. Finally, each of the four workers outputs a batch of sample data, which the system can then aggregate for subsequent model training. In this way, by utilizing multiple independent environment instances and parallel workers, four times the sample volume of a single environment can be collected per unit time, significantly improving data acquisition efficiency. It should be understood that the foregoing description is merely illustrative and does not constitute a limitation of this disclosure.

[0061] As a refinement of the above embodiments, when performing step 103, which involves parallel gradient calculation of the sample data acquired by each of the sub-training nodes based on the second preset number of sub-training nodes to obtain the local gradients corresponding to the agent and the sample data of different batches, the following implementation methods can also be adopted, but are not limited to: loading the policy network copy and value network copy of the agent into each of the second preset number of sub-training nodes; inputting the sample data corresponding to each sub-training node into the policy network copy and value network copy in the corresponding sub-training node, and calculating the policy gradient loss and value loss corresponding to each sub-training node; calculating the total loss corresponding to each sub-training node based on the policy gradient loss and value loss corresponding to each sub-training node; and calculating the local gradients corresponding to the agent and the sample data of different batches based on the total loss corresponding to each sub-training node.

[0062] To facilitate understanding of the above embodiments, this embodiment provides an exemplary description, including: setting the second preset number of sub-training nodes to 3 ( ), and set the training objectives for the agent, with each sub-training node loading a copy of the agent's policy network ( ) and value network copy ( ), where θ and φ are the parameters of the policy network and the value network, respectively. Three sub-training nodes each deploy a replica of the network with identical parameters: load and ; load and ; load and All replicas' initial parameters are satisfied. (Main network parameters) (Main network parameters), but gradients are stored independently in subsequent calculations. Each node obtains different batches of samples. Each sample contains (state s, action a, reward r, next state s'); deal with Policy Network Calculate policy gradient loss The formula is:

[0063]

[0064] in, It represents the policy gradient loss, which measures the quality of the action selection of the policy network. This represents the expected value (mean) of a batch of samples. Represents the policy network (parameters are) The probability of choosing action a in state s; Represents the logarithm of the policy probability (used for gradient calculation); The advantage function represents the relative value of action a in state s, and the subscript 1 corresponds to the calculation of the first batch of sample data.

[0065] Value Network Calculate value loss The formula is:

[0066]

[0067] in, γ represents the value loss, measuring the prediction error of the value network; r represents the immediate reward obtained by performing action a in the current state s; γ represents the discount factor. This is used to decay the weight of future rewards; Represents the value network (parameter is) Predicting the value of the current state s; This represents the value prediction of the value network for the next state s′; This represents the target value (current reward + next state discount value).

[0068] The formula for calculating the total loss is:

[0069]

[0070] in, This represents the total loss, which combines strategic loss and value loss. Represents the balance coefficient Adjust the weights of the two types of losses.

[0071] Similarly, Node2 processes batch_002: The calculation is performed in the same way. , Total loss ;

[0072] Node3 processes batch_003: The calculation is done similarly. , Total loss .

[0073] Each sub-training node calculates its local gradient by taking the partial derivative of the total loss with respect to the network parameters. The local gradient for Node1 includes: (Policy network parameter gradient) (Value network parameter gradient); The local gradients for Node2 include: , ; The corresponding local gradients include: , These local gradients reflect the parameter adjustment requirements of each node for its own sample data (e.g., Indicated as reduction Corresponding total loss, policy network parameters (The direction and magnitude to be adjusted). Subsequently, the global gradient can be obtained through gradient aggregation (e.g., averaging) to update the main network parameters. and .

[0074] As a refinement of the above embodiments, when performing step 104, which involves aggregating all the local gradients based on the parameter server to obtain the global gradient of the agent, the following implementation methods may also be adopted, for example: receiving the local gradients corresponding to the agent and different batches of sample data based on the parameter server; and calculating the weighted average of all received local gradients to obtain the global gradient.

[0075] To facilitate understanding of the above embodiments, this embodiment provides an exemplary description of global gradient calculation, including:

[0076] The formula for calculating the global gradient weighted average of the policy network is:

[0077]

[0078] Where ▽θglobal represents the global gradient of the policy network; w i ▽θ represents the weight of the i-th sub-training node (usually related to the sample size); i This represents the local gradient of the policy network calculated at the i-th sub-training node.

[0079] The formula for calculating the global gradient weighted average of the value network is:

[0080]

[0081] Where ▽ϕglobal represents the global gradient of the value network; w i Represents the weight of the i-th sub-training node (usually related to the sample size); ▽ϕ i This represents the local gradient of the value network calculated at the i-th sub-training node.

[0082] By using the aforementioned global gradient calculation formula, the global gradient of the value network and the global gradient of the policy network corresponding to the agent are calculated. The global parameters of the agent are then modified using these two types of parameters to obtain the trained agent.

[0083] As a refinement of the above embodiments, the method may also adopt, but is not limited to, the following implementation methods. For example, if the parameter server adopts a synchronous aggregation method in the process of aggregating the local gradients, then based on a first preset number of data acquisition workers, sample data of the agent's interaction with the environment are collected in parallel; based on a second preset number of sub-training nodes, different batches of the sample data are acquired in parallel; and based on the second preset number of sub-training nodes, gradient calculations are performed in parallel on the sample data acquired by each of them to obtain the local gradients corresponding to the agent and the sample data of different batches. These three steps are executed in parallel. Based on the parameter server, all the local gradients are aggregated to obtain the global gradient of the agent. This step is executed independently of the three steps. If the parameter server adopts an asynchronous aggregation method, then the three steps are executed in parallel with this step. The synchronous aggregation method is to perform local gradient aggregation when it is determined that the parameter server has received all local gradients. The asynchronous aggregation method is to perform local gradient aggregation when it is determined that the parameter server has received any number of local gradients.

[0084] As a refinement of the above embodiments, after obtaining the trained agent, the method may also adopt, but is not limited to, the following implementation methods, for example: displaying the usage effect of the trained agent based on the model demonstration class; evaluating the usage effect based on the model evaluation class to obtain the performance evaluation index of the agent, wherein the model evaluation class and the model demonstration class are both decorated with the decorator and executed in parallel.

[0085] In one possible implementation of this application, the methods involved in reinforcement learning are divided into the following modules: env: generates a gym environment based on the ID and calculates the environment reward; evaluator: evaluates the performance of the currently trained model; exploration: explores and balances between known optimal policies and unknown policies, commonly using methods such as e-greedy exploration, sampling policies, and state search; learner: calculates the model gradient based on the sampled data; parameter_server: a parameter server that stores the latest model parameters; rollout: interacts with the environment to generate training data; replay_buffer: stores training data from which the learner samples to train the model; tensorboard: displays the model training effect; trainer: the overall training process, calculates gradients through the learner, uploads gradients periodically, and retrieves model parameters periodically. The aforementioned modules are decorated with `ray.remote` to achieve parallel execution of all steps of reinforcement learning.

[0086] Figure 2A schematic diagram of a ray-based reinforcement learning training framework provided in an embodiment of this application. Figure 2 The diagram shows two data acquisition workers, two replay buffers, two sub-training nodes, and one parameter server. Their data transmission relationships are illustrated in the diagram. It is important to understand that... Figure 2 The module structure shown does not constitute a limitation of this application.

[0087] In summary, the embodiments of this application can achieve the following effects, including:

[0088] By setting a specified number of data acquisition workers to concurrently collect training data, configuring multiple sub-training nodes to sample and generate gradients, uploading gradient data at fixed time intervals, and fetching the latest model parameters, all these steps can be executed concurrently. Furthermore, Ray dynamically uses machine resources according to demand, maximizing the utilization of single server resources and improving training efficiency.

[0089] The above are embodiments of the method proposed in this application. Based on the same inventive concept, embodiments of this application also provide a reinforcement learning training device based on the ray framework, the structure of which is as follows: Figure 3 As shown.

[0090] Figure 3 This is a schematic diagram of the internal structure of a reinforcement learning training device based on the ray framework, provided as an embodiment of this application. Figure 3 As shown, the device includes:

[0091] At least one processor 301;

[0092] And a memory 302 that is communicatively connected to at least one processor;

[0093] The memory 302 stores instructions executable by at least one processor. These instructions are executed by at least one processor 301 to enable the processor 301 to: acquire sample data of the agent's interaction with the environment in parallel using a first preset number of data acquisition workers, each of which is decorated using a ray framework decorator; acquire different batches of the sample data in parallel using a second preset number of sub-training nodes, where the second preset number and the first preset number are subject to preset constraints during training, and each of the second preset number of sub-training nodes is decorated using the decorator; perform gradient calculations in parallel on the sample data acquired by each of the second preset number of sub-training nodes to obtain the local gradients corresponding to the agent and the different batches of sample data; and aggregate all the local gradients using a parameter server to obtain the global gradient of the agent, thereby updating the global parameters of the agent based on the global gradient to obtain the trained agent.

[0094] Some embodiments of this application provide corresponding to Figure 1 A non-volatile computer storage medium stores computer-executable instructions, which are configured to: collect sample data of an agent interacting with its environment in parallel using a first preset number of data acquisition workers, each of which is decorated using a ray frame decorator; acquire different batches of the sample data in parallel using a second preset number of sub-training nodes, wherein the second preset number and the first preset number are subject to preset constraints during training, and each of the second preset number of sub-training nodes is decorated using the decorator; perform gradient calculations in parallel on the sample data acquired by each of the second preset number of sub-training nodes to obtain local gradients corresponding to the agent and different batches of the sample data; aggregate all the local gradients using a parameter server to obtain the global gradient of the agent, and update the global parameters of the agent according to the global gradient to obtain the trained agent.

[0095] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments for IoT devices and media are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0096] The systems, media, and methods provided in this application are one-to-one correspondences. Therefore, the systems and media also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the systems and media will not be repeated here.

[0097] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0098] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0099] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0100] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0101] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0102] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0103] Computer-readable media include both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0104] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0105] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for reinforcement learning training based on a ray framework, characterized in that, The method comprises: collecting sample data of interaction between an agent and an environment in parallel based on a first preset number of data collection workers, each of the first preset number of data collection workers being modified by a decorator of a ray framework; acquiring different batches of the sample data in parallel based on a second preset number of sub-training nodes, the second preset number and the first preset number having a preset constraint condition in a training process, each of the second preset number of sub-training nodes being modified by the decorator; performing gradient calculation on the sample data acquired by each of the second preset number of sub-training nodes in parallel to obtain local gradients of the agent corresponding to different batches of the sample data; aggregating all the local gradients based on a parameter server to obtain a global gradient of the agent, updating a global parameter of the agent according to the global gradient to obtain a trained agent. 2.The ray framework-based reinforcement learning training method of claim 1, wherein, After acquiring different batches of the sample data in parallel based on the second preset number of sub-training nodes, the method further comprises: setting different batch IDs for different batches of the sample data respectively; binding different cache threads to different batch IDs respectively; when determining that a sample data quantity of any batch of the sample data of different batches meets a preset quantity threshold, caching the sample data of the batch to a replay cache area based on the cache thread bound to the batch ID, and performing the caching task of the sample data in parallel between the different cache threads.

3. The method of claim 2, wherein the ray framework is based on a reinforcement learning training method. The acquiring different batches of the sample data in parallel based on the second preset number of sub-training nodes comprises: uniformly matching different batches of the sample data in the replay cache area to the second preset number of sub-training nodes; acquiring the sample data matched to each of the second preset number of sub-training nodes from the replay cache area in parallel.

4. The reinforcement learning training method based on a ray framework according to claim 1, characterized in that, The collecting sample data of interaction between an agent and an environment in parallel based on a first preset number of data collection workers comprises: generating a plurality of independent environment instances based on a Gym environment interface; loading the plurality of independent environment instances by the first preset number of data collection workers respectively, so that each of the first preset number of data collection workers executes an interaction strategy of the agent and the environment instance loaded thereby to obtain corresponding sample data, any data collection worker of the first preset number of data collection workers loads one environment instance of the plurality of independent environment instances, and the environment instances loaded by any two of the data collection workers are different. 5.The ray framework-based reinforcement learning training method of claim 1, wherein, The performing gradient calculation on the sample data acquired by each of the second preset number of sub-training nodes in parallel to obtain local gradients of the agent corresponding to different batches of the sample data comprises: loading a policy network copy and a value network copy of the agent in each of the second preset number of sub-training nodes; inputting the sample data corresponding to each sub-training node into the policy network copy and the value network copy in the corresponding sub-training node to calculate a policy gradient loss and a value loss corresponding to each of the sub-training nodes respectively; and The total loss corresponding to each sub-training node is calculated based on the policy gradient loss and the value loss corresponding to the sub-training node; The local gradient corresponding to the agent and the sample data of different batches is calculated based on the total loss corresponding to each sub-training node.

6. The reinforcement learning training method based on a ray framework according to claim 5, characterized in that, The global gradient of the agent is obtained by aggregating all the local gradients based on the parameter server, including: The parameter server receives the local gradient corresponding to the agent and the sample data of different batches; All the received local gradients are weighted and averaged to obtain the global gradient.

7. The ray framework-based reinforcement learning training method according to any one of claims 1-6, characterized in that, The method includes: If the parameter server adopts a synchronous aggregation method in the process of aggregating the local gradient, a first preset number of data collection workers are used to collect sample data of agent and environment interaction in parallel, a second preset number of sub-training nodes are used to obtain the sample data of different batches in parallel, and the second preset number of sub-training nodes are used to perform gradient calculation on the sample data obtained by each sub-training node in parallel to obtain the local gradient corresponding to the agent and the sample data of different batches. The three steps are executed in parallel, and the global gradient of the agent is obtained by aggregating all the local gradients based on the parameter server, which is independent of the execution of the three steps; If the parameter server adopts an asynchronous aggregation method, the three steps and the step are executed in parallel. The synchronous aggregation method is to perform local gradient aggregation when it is determined that the parameter server has received all the local gradients. The asynchronous aggregation method is to perform local gradient aggregation when it is determined that the parameter server has received any number of local gradients. 8.The ray framework based reinforcement learning training method of claim 7, wherein, After obtaining the trained agent, the method further includes: The use effect of the trained agent is displayed based on a model display class; The performance evaluation index of the agent is obtained by evaluating the use effect based on a model evaluation class. The model evaluation class and the model display class are both modified by the decorator and executed in parallel. 9.A reinforcement learning training device based on a ray framework, characterized in that, The device includes: At least one processor; and a memory connected in communication with the at least one processor; The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the reinforcement learning training method based on the ray framework as claimed in any one of claims 1-8.

10. A computer storage medium storing computer-executable instructions, which, when executed by a processor, cause the processor to perform acts comprising: The computer executable instructions, when executed, implement the reinforcement learning training method based on the ray framework as claimed in any one of claims 1-8.

Images