Large-scale reinforcement learning training framework system based on distributed design
By running multiple actors and learner modules in a distributed and parallel manner within the cluster, and using multiple computers and GPU resources, the problems of low data utilization rate and single-machine resource limitation of deep reinforcement learning algorithms are solved, and efficient reinforcement learning training is achieved. Shorten training time and reduce costs.
Patent Information
- Application Number
- CN202210612321.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-31
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2042-05-31
AI Technical Summary
The current deep reinforcement learning algorithm has low data utilization, long training time and high cost, especially in complex tasks and real scenarios, and the application is limited, and the stand-alone CPU resources limit the data acquisition amount and training efficiency.
Design a large-scale reinforcement learning training framework based on distributed design, and improve data acquisition and utilization efficiency by running multiple actor modules and learner modules in distributed parallel within the cluster.
Large-scale, high-concurrency reinforced learning training is realized, shortening the time for training agents, improving data generation and utilization efficiency, reducing training costs, and meeting the ease of use of users.
Smart Images

Figure CN114861826B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a large-scale reinforcement learning training framework system based on distributed design. Background Art
[0002] In recent years, more and more researchers in the field of artificial intelligence have turned to the study of reinforcement learning. In 2013, the DQN model proposed by DeepMind combined deep neural networks with traditional Q-learning reinforcement learning algorithms for the first time, surpassing the level of human experts in Atari games. Since then, people's research on reinforcement learning has entered the era of deep reinforcement learning. From 2016 to 2017, DeepMind successively developed AlphaGo and AlphaGo Zero Go AI, which pioneered the combination of deep reinforcement learning and Monte Carlo tree search, and defeated top human professional chess players many times, marking a major breakthrough in deep reinforcement learning for complex problems. In the following two years, deep reinforcement learning was used to conquer the most difficult video games today, Dota 2 and StarCraft 2, both of which defeated top human players with overwhelming advantages and successfully stood at the top of game AI.
[0003] Reinforcement learning is a discipline that solves sequential decision-making problems, and reinforcement learning agents are AIs that are trained through relevant algorithms and can be used to solve such problems. For example, we can use reinforcement learning algorithms to train an agent that is proficient in games; we can also use similar methods to train a robot that can help logistics companies dispatch goods. At the same time, for each decision made by the reinforcement learning agent, such problems need to be judged by a suitable judgment mechanism - when the agent does it right, we need to "reward"; when the agent's decision is not good, we need to "punish". Therefore, sequentiality and judgeability constitute two important elements of the decision-making problems studied by reinforcement learning. On the other hand, such problems can be formally defined as Markov decision process problems, namely MDP problems.
[0004] Specifically, the MDP problem can be expressed in language as follows: the agent is in the current state s of the environment t , make a suitable action a with some strategy π t , the environment then deduces the state s at the next moment based on the actions taken by the agent t+1 , and returns a reward value r to the agent t+1 , so far one round of iteration is completed. The agent and the environment continue to interact as described above and accumulate the reward value r obtained at each moment. The goal of reinforcement learning is to maximize the cumulative reward value ∑ of the agent in the decision-making process t r tFrom the perspective of data acquisition sources, the data distribution of most machine learning algorithms is irrelevant to the quality of the model, and the relevant training data is also collected through a series of offline methods. In reinforcement learning problems, since the training data is generated through the above-mentioned interactive form, the training results of the model will in turn affect the distribution of the data, so data always needs to be collected online through the interaction between the model and the environment.
[0005] While collecting the above data, researchers will use this data to train specific models. After entering the era of deep learning, these models have changed from simple linear models to models based on deep neural networks. The new models obtained through training will be given to intelligent agents, allowing them to make better decisions, so that they can collect better data through continuous interaction with the environment. This completes a complete set of training iterations for reinforcement learning.
[0006] The current solution mainly has the following two types of technical problems:
[0007] Technical Problem 1: Current deep reinforcement learning algorithms suffer from the problem of low data utilization, requiring the agent to collect a large amount of training data in order to train a better model. For example, in Atari games, training the DQN algorithm requires millions or even tens of millions of interactions with the environment, and the training time is as long as several days. For the more complex game Dota 2, AI has to spend the equivalent of 45,000 years of human time on self-game reinforcement learning. In contrast, humans only need a few weeks to master a game through learning. The low data utilization efficiency increases the training cost of deep reinforcement learning and limits its feasibility in applying it to complex tasks and real-world scenarios.
[0008] Technical Problem 2: Since it is difficult to use GPU for computing acceleration in environmental simulation programs, most of the current simulation environments run on CPUs. The limited CPU resources of a single machine directly limit the unit data collection volume of the single machine reinforcement learning algorithm, increasing the time cost of single machine training.
[0009] In view of this, it is necessary to study a new training framework to break through the physical limitations of single-machine training and thus improve the efficiency of data collection and utilization. Summary of the invention
[0010] The purpose of the present invention is to provide a large-scale reinforcement learning training framework system based on distributed design, which can improve the efficiency of data collection and utilization.
[0011] The objective of the present invention is achieved through the following technical solutions:
[0012] A large-scale reinforcement learning training framework system based on distributed design includes: N actor modules and M learner modules that are deployed on multiple computers in a cluster and run in parallel and distributed, and a training data centralized storage module. The actor modules and learner modules occupy different computing resources in their respective computers during operation, and N and M are both set positive integers; wherein:
[0013] Each actor module is used to obtain training data from the current application scenario through the network model currently contained in the agent;
[0014] A training data set centralized storage module is used to store training data;
[0015] Each learner module is used to extract training data from the storage module in the training data set to train the network model, and output the trained network model parameters to update the network model included in the intelligent agent in the actor module.
[0016] It can be seen from the technical solution provided by the present invention that the present invention provides a new distributed reinforcement learning training framework, which improves the training efficiency of the reinforcement learning process through the efficient use of distributed cluster resources, realizes large-scale and high-concurrency reinforcement learning training, shortens the time required for researchers to train reinforcement learning agents, and at the same time meets the ease of use for users. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.
[0018] Figure 1 A schematic diagram of a large-scale reinforcement learning training framework system based on distributed design provided by an embodiment of the present invention;
[0019] Figure 2 A schematic diagram of a single actor module provided for an embodiment of the present invention;
[0020] Figure 3 Schematic diagram of the interaction between the reinforcement learning agent and the environment provided by the embodiment of the present invention
[0021] Figure 4 A schematic diagram of a single learner module provided by an embodiment of the present invention;
[0022] Figure 5 A schematic diagram of a request-response mode provided by an embodiment of the present invention;
[0023] Figure 6 A schematic diagram of a publisher-subscriber model provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0024] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of the present invention.
[0025] First, the terms that may be used in this article are explained as follows:
[0026] The terms "include", "comprises", "contains", "has" or other descriptions with similar semantics should be interpreted as non-exclusive inclusion. For example, including certain technical feature elements (such as raw materials, components, ingredients, carriers, dosage forms, materials, dimensions, parts, components, mechanisms, devices, steps, procedures, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products or products, etc.) should be interpreted as including not only certain technical feature elements explicitly listed, but also other technical feature elements known in the art that are not explicitly listed.
[0027] The following is a detailed description of a large-scale reinforcement learning training framework system based on distributed design provided by the present invention. The contents not described in detail in the embodiments of the present invention belong to the prior art known to professional and technical personnel in the field. If the specific conditions are not indicated in the embodiments of the present invention, they are carried out according to the conventional conditions in the field or the conditions recommended by the manufacturer. The reagents or instruments used in the embodiments of the present invention that do not indicate the manufacturer are all conventional products that can be purchased commercially.
[0028] The embodiment of the present invention provides a large-scale reinforcement learning training framework system based on distributed design, which generates training data in a distributed and parallel manner on multiple machines in a cluster in the form of multiple actors; at the same time, in the form of multiple learners, the model is trained in a distributed and parallel manner on multiple GPU cards, thereby making full use of the computing resources of a large-scale cluster and shortening the time required for researchers to train reinforcement learning agents. In other words, the present invention mainly makes improvements at the training framework level, and the rest of the training processes and network models involved can refer to conventional technologies, and the present invention will not elaborate on them.
[0029] Figure 1The structure of the training framework system provided by the present invention is shown, which mainly includes: N actor modules and M learner modules that are deployed on multiple computers in a cluster and run in parallel in a distributed manner, a training data centralized storage module, and a communication module; wherein:
[0030] Each actor module is used to obtain training data from the current application scenario through the network model currently contained in the agent.
[0031] The training data set centralized storage module is used to store training data.
[0032] Each learner module is used to extract training data from the storage module in the training data set to train the network model, and output the trained network model parameters to update the network model included in the intelligent agent in the actor module.
[0033] Communication module: The interaction between the above modules needs to be connected through the communication module, which is mainly responsible for transmitting two types of resources: training data and network models.
[0034] In the embodiment of the present invention, the actor module and the learner module occupy different computing resources in the computer they belong to when they are running. Specifically, each actor module occupies a CPU core in the computer they belong to as a process; each learner module occupies a GPU card in the computer they belong to. N and M are both set positive integers, and the specific values can be set according to actual conditions or experience.
[0035] To facilitate understanding, each part of the system is introduced in detail below.
[0036] 1. Actor module.
[0037] Figure 2 The main structure of a single action module is shown, including: network model reasoning acceleration module, parallel environment deduction module, and the first communication submodule of the network model discovery module; among them:
[0038] The network model reasoning acceleration module includes n agent copies, and reasoning is performed through the network model in the agent copy, and the reasoning speed of the network model is improved as much as possible, where n is a set positive integer. The specific structure of the network model can be set according to the actual situation. The structure of the network model in the n agent copies is exactly the same, and the latest model parameters will be used for each update. However, since the timing of updating parameters of the network models in different agent copies is different (asynchronous updating of their own parameters), there will be differences in parameters.
[0039] The parallel environment deduction module contains n environment copies, which are responsible for maintaining a set of "agent-environment" interactions in parallel. There is no difference in the environment configuration of the n environment copies, but there are differences in the environment content (environment state); taking the game scene as an example, when the same game is started using n identical smart terminals, each game environment corresponds to an environment copy. Since the decision of each smart terminal (which can be understood as the action output by the network model) is random, there is randomness in the game environment. The states of the n games after one minute are different. Therefore, although the environment copies are the same in the initial stage, the environment states will differ with continuous interaction.
[0040] The network model discovery module is used to search whether there is a trained network model. If so, the network models in all agent copies are replaced with the newly trained network model.
[0041] The first communication submodule (not shown in the figure) is used to establish a communication connection with the communication module to transmit training data and model parameters.
[0042] In the embodiment of the present invention, the training data is obtained through the interaction between the network model reasoning acceleration module and the parallel environment deduction module and through a set preprocessing method, the steps include:
[0043] (1) Through one-to-one interaction between n agent copies and n environment copies, the environmental state (s1, s2, …, s n ), and the decision information (a1, a2, …, a n ), where s i represents the state of the environment of the i-th environment copy at time t, a i Represents the decision information generated by the network model currently contained in the i-th agent copy, i = 1, 2, ..., n;
[0044] (2) The decision information (a1, a2, ..., a n ) is returned to the corresponding environment copy, and through interaction, the environment state (s′1, s′2, …, s′) of n environment copies at time t+1 is obtained. n ) and reward; where s′ i Represents the environment state of the i-th environment copy at time t+1.
[0045] like Figure 3 As shown in Figure 1, it is a schematic diagram of the interaction between the reinforcement learning agent and the environment. Agent represents the agent, Environment represents the environment, and state S trepresents the environment state (s1, s2, ..., s n ), actionA t It indicates the decision information (a1, a2, ..., a1) generated by the network models currently contained in the n agent copies according to the corresponding environment state. n ), rewardR t Indicates that actionA t The return (i.e., reward) obtained by returning to the corresponding environment copy; similarly, S t+1 With R t+1 Represents the state of the environment at time t+1 of n environment copies, and the reward obtained after returning the corresponding decision information to the environment copy.
[0046] (3) Repeat the interaction multiple times, that is, repeat the above steps 1) to 2), and then process the environment state, or the environment state, decision information and reward through the set preprocessing method to obtain training data; the training data will be sent to the training data set storage module for storage. After the sending is completed, the actor module continues to execute the above steps 1) to 3) until the training is completed. At the same time, after the training data is sent, the network model discovery module will periodically search for newly trained network model parameters. If so, the parameters in all agent copies will be replaced with the new network model parameters.
[0047] In the embodiment of the present invention, the environment refers to the application scenario of reinforcement learning. If the application scenario is a game scenario, the environment is the game scenario. In the game scenario, the environmental state includes: the state of each object that can be observed in the game's image interface, for example, the state of the character controlled by the player, and the state of the inherent objects in the environment (objects that the player cannot control). Decision information is the action taken by the control character. Taking the game of Honor of Kings as an example, the environmental state includes: 1) The state of the character controlled by the player that can be observed: the state of the hero in the field of vision, for example, one or more of the information such as blood volume, blue volume, money, equipment, and position; 2) The state of the inherent objects in the environment (objects that the player cannot control): the state of the soldiers and wild monsters (position and blood volume, etc.), terrain (the location and size of obstacles, etc.), the state of the tower (blood volume and whether it is protected), etc. Decision information includes: action instructions, which are used to control the actions taken by the hero in the current environmental state, for example, moving or releasing specific skills. When the decision information is returned to the environment, the environmental state will change. For example, if the decision information is to move or release a specific skill, the location of the corresponding impact of the environmental state will change; or the mana, health of an enemy hero or soldier will change.
[0048] Those skilled in the art can understand that the purpose of reinforcement learning is to obtain the best decision in a selected environment. The information involved mainly includes: 1) state space, for example, the set of all states contained in the game of Honor of Kings introduced above; 2) action space, for example, the set of all actions contained in the game of Honor of Kings introduced above; 3) reward function: a function used to describe "the reward that can be obtained by performing a specific action (decision) in a specific state". 4) state transition function: a function used to describe "the next state that the environment will deduce after performing a specific action in a specific state".
[0049] In the embodiment of the present invention, the preprocessing is divided into the preprocessing of the environmental state and the processing of converting the decision information and rewards into data available for training. The preprocessing methods for different environmental states are different. In the Atari game, the game area of the video is cropped, the resolution of the image is reduced, and grayscale processing is performed. The processing of the King of Glory game includes: onehot encoding of the category data, etc. The processing of converting decision information and rewards into data available for training is related to the specific training scheme. If the PPO (Proximal Policy Optimization) algorithm is selected, GAE (Generalized Advantage Estimation) needs to be calculated from the data as part of the training data. If the DQN algorithm is selected, the data obtained by the preprocessing of the environmental state can be directly applied without further processing. In summary, the training data obtained after preprocessing includes: the preprocessed environmental state obtained through the scene image, and the decision information and rewards available for training after preprocessing. .
[0050] It should be noted that the above-mentioned PPO algorithm and DQN algorithm are both algorithms involved in training the network model of the learner module. The specific algorithm process can be referred to conventional technology, which will not be elaborated in the present invention. Of course, the above two types of algorithms are only examples and do not constitute a limitation. In actual applications, users can choose the algorithm of the training stage according to their needs or experience.
[0051] 2. Learner module.
[0052] Figure 4 The main structure of a single learner module is shown, including: training data pre-fetching module, network model optimization module, network model synchronization module second communication submodule; among which:
[0053] The training data pre-fetching module is used to asynchronously pre-fetch the training data into the video memory of the GPU during training to improve the training efficiency.
[0054] The network model optimization module is used to train the network model using the training data to obtain the parameters of the optimized network model. The network model and training method used are specified by the user and are not limited by the present invention.
[0055] The network model synchronization module is used to synchronize the network model parameters using the parameters of the optimized network model obtained by all learner modules when the number of learner modules M is greater than 1. For example, the ring AllReduce algorithm can be used to synchronize the network model parameters. If M=1, the parameters of the optimized network model obtained by the network model optimization module will be directly synchronized to all actors by the communication module.
[0056] The second communication submodule (not shown in the figure) is used to establish a communication connection with the communication module to transmit training data and model parameters.
[0057] 3. Training data set centralized storage module.
[0058] In the embodiment of the present invention, the training data set storage module is mainly used to save the training data generated by the actor and provide it for the learner to use for training. Specifically: a circular queue can be used to store the training data, and it includes two threads, which are responsible for receiving the training data and storing it in the queue, and extracting the training data for the learner to train. The threads are controlled by conditional locks to achieve the atomicity of queue operations.
[0059] 4. Communication module.
[0060] In the embodiment of the present invention, the communication module is mainly used to transmit training data and parameters of the network model.
[0061] The communication module is implemented by a message queue. After startup, it mainly performs the following two communication modes:
[0062] The first communication mode is the request-reply (Req-Rep) mode, such as Figure 5 As shown, this mode is similar to the communication between the client and the server. It is a many-to-one transmission mode, which is responsible for sending the training data obtained by the actor module to the storage module in the training data set;
[0063] The second communication mode is the publisher-subscriber (Pub-Sub) mode, such as Figure 6 As shown in Figure 1, this mode is a one-to-many transmission mode, in which the learner module publishes the trained network model parameters to all actor modules that subscribe to the model through broadcast communication.
[0064] Preferably, in order to facilitate user use, the above four types of modules should be appropriately hidden and transparent to users. Therefore, the present invention also designs a set of user interfaces to facilitate researchers to use the framework more simply without knowing most of the implementation details. Mainly including:
[0065] (1) Network model interface, which is used to encapsulate the construction, inference, parameter storage and reading of network models.
[0066] In the embodiment of the present invention, an interface design that is independent of the deep learning framework is used, and the user can select any framework, such as Tensorflow and PyTorch, as an implementation tool for the network model.
[0067] (2) Agent interface, which is used to encapsulate the main process of reinforcement learning.
[0068] In the embodiment of the present invention, data collection (Sample) in the actor module and model training (Learn) in the learner module are implemented under the same agent class, so that users can easily ensure the consistency of each process of the algorithm. In addition, the agent class also encapsulates functions such as state preprocessing, training data post-processing, and hyperparameter modification.
[0069] (3) Environmental interface, used to encapsulate various environmental information.
[0070] In the embodiment of the present invention, the environment interface adopts a design similar to OpenAIGym. In addition to retaining some interfaces in Gym, it also encapsulates the preprocessing of the state in the environment and the design of reward value (RewardShaping). Users can easily access the new research environment.
[0071] In addition, for the agent interface and model interface, the present invention uses a unified hyperparameter configuration interface and metaprogramming method to easily solidify and read the experimental configuration. After the user defines the algorithm and model, the framework system will automatically identify the hyperparameters therein, and save the specific values of the corresponding hyperparameters when the framework system is running, and record them as configuration files. The framework system can read the configuration file and use the same configuration to reproduce the experiment.
[0072] The main working process of the above training framework system provided by the present invention is as follows:
[0073] (1) Configure a series of data such as network models, intelligent agents, and environments through the user interface.
[0074] (2) When training starts, the actor module, learner module, training data centralized storage module and communication module are started successively. The actor module will start multiple copies on multiple machines. If each machine contains multiple CPU cores, each actor module will occupy one CPU core of the current machine and start as a process. When there are multiple learner modules participating in the training, each learner occupies a GPU card and is also started on multiple machines. In order to transfer data, each actor module and learner module will start the corresponding communication submodule to establish a communication connection with the communication module in the system. In addition, the centralized storage module will not start multiple copies.
[0075] (3) After each actor is started, it first initializes n copies of the agent and an equal number of copies of the environment. Then it starts to loop through network model reasoning and parallel environment deduction. After repeating the network model reasoning and parallel environment deduction a certain number of times, it uses the data processing method specified by the user to obtain training data and send it to the centralized storage module. After sending, the actor continues to execute the above loop until the training is completed. At the same time, after sending the training data, the actor module will periodically search through the network model discovery module to see if there is a newly trained model. If so, the model in all agent copies will be replaced with the new model.
[0076] (4) After each learner is started, it initializes an agent copy to save and train the model. After that, it pre-fetches training data and optimizes the network model. If there is only one learner, the optimized model parameters will be directly synchronized to all actors by the communication module. When multiple learners participate in distributed training, it is necessary to synchronize the network model parameters through the network model synchronization module using the ring AllReduce algorithm.
[0077] The above steps (3) to (4) are continuously repeated until the training stop condition is met, for example, the number of training times reaches the target.
[0078] The above-mentioned training framework system provided by the embodiment of the present invention can generate training data on a large scale and with high concurrency by running multiple actor modules in a distributed manner and utilizing a large amount of CPU computing resources of a cluster, thus breaking through the physical limitations of single-machine training and greatly improving the efficiency of data generation in reinforcement learning. By supporting multiple learner modules to join the training, this framework further improves the training efficiency. At the same time, in order to facilitate user use, the training framework system performs clear and reasonable abstraction and designs interfaces for all processes in reinforcement learning, so that users do not need to know most of the implementation details involved in the training. Therefore, through the large-scale reinforcement learning distributed training framework system designed by the present invention, reinforcement learning researchers can conduct large-scale reinforcement learning more easily, make full use of the computing resources of large-scale clusters, and shorten the time required to train intelligent agents.
[0079] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.
Claims
1. A large-scale reinforcement learning training framework system based on distributed design, characterized in that: include: N actor modules and M learner modules are deployed on multiple computers in the cluster and run in a distributed and parallel manner, as well as a training data set centralized storage module. The actor modules and learner modules occupy different computing resources in their respective computers during operation. N and M are both set positive integers; where: Each actor module is used to obtain training data from the current application scenario through the network model currently contained in the agent; A training data set centralized storage module is used to store training data; Each learner module is used to extract training data from the storage module in the training data set to train the network model, and output the trained network model parameters to update the network model included in the intelligent agent in the actor module; Each actor module includes: a network model reasoning acceleration module, a parallel environment deduction module and a network model discovery module; wherein: the network model reasoning acceleration module includes n agent copies, the parallel environment deduction module includes n environment copies, and the n environment copies are all application scenarios; the network model discovery module is used to search whether there is a trained network model, and if so, the network models in all agent copies are replaced with the newly trained network model; wherein n is a set positive integer; the training data is obtained through the interaction between the network model reasoning acceleration module and the parallel environment deduction module and through the set preprocessing method; The steps of obtaining training data through the interaction between the network model reasoning acceleration module and the parallel environment deduction module and through a set preprocessing method include: Through one-to-one interaction between n agent copies and n environment copies, the environment states (s1, s2, ..., s n ), and the decision information (a1, a2, ..., a n ), where s i represents the state of the environment of the i-th environment copy at time t, a i represents the decision information generated by the network model currently contained in the i-th agent copy, i=1, 2, ..., n; the environment state includes: various states of various objects presented in the image of the application scene; the decision information includes: action instructions for a certain role in the application scene; The decision information (a1, a2, ..., a n ) is returned to the corresponding environment copy, and through interaction, the environment state (s′1, s′2, ..., s′) of n environment copies at time t+1 is obtained. n ) reward; where s′ i Represents the environmental state of the i-th environmental copy at time t+1; After repeated interactions, the environment state, or the environment state, decision information and rewards are processed through the set preprocessing method to obtain training data.
2. According to the large-scale reinforcement learning training framework system based on distributed design according to claim 1, it is characterized in that: The actor module and the learner module respectively occupy different computing resources in the computer when running, including: each actor module occupies a CPU core in the computer as a process; each learner module occupies a GPU card in the computer.
3. A large-scale reinforcement learning training framework system based on distributed design according to claim 1 or 2, characterized in that: Each learner module includes: a training data pre-fetching module, a network model optimization module and a network model synchronization module; wherein: A training data pre-fetching module is used to asynchronously pre-fetch training data into the video memory of the GPU card during training; The network model optimization module is used to train the network model using the training data to obtain the parameters of the optimized network model; The network model synchronization module is used to synchronize the network model parameters using the parameters of the optimized network models obtained by all learner modules when the number M of learner modules is greater than 1.
4. According to the large-scale reinforcement learning training framework system based on distributed design according to claim 1, it is characterized in that: The system also includes: a communication module for transmitting training data and a network model; Each actor module and each learner module includes a corresponding communication submodule, which is used to communicate with the communication module to transmit training data and model parameters.
5. According to claim 4, a large-scale reinforcement learning training framework system based on distributed design is characterized in that: The communication module includes two communication modes: The first communication mode is the request-reply mode, which is a many-to-one transmission mode and is responsible for sending the training data obtained by the actor module to the storage module in the training data set; The second communication mode is the publisher-subscriber mode, which is a one-to-many transmission mode. The learner module publishes the trained network model parameters to all actor modules that subscribe to the model through broadcast communication.
6. According to a large-scale reinforcement learning training framework system based on distributed design according to claim 1, it is characterized in that: The system also includes: a user interface; the user interface includes: Network model interface, used to encapsulate the construction, reasoning, parameter saving and reading of network models; Agent interface, used to encapsulate the main process of reinforcement learning; Environment interface, used to encapsulate various types of environment information.
