A decision-making control method for autonomous equipment based on distributed reinforcement learning
Through distributed reinforcement learning technology, a training system including sampling nodes, cache nodes and training nodes was established, which solved the problem of limited computing resources of autonomous equipment decision-making control methods in the existing technology, and achieved efficient training in complex problem situations and The cost reduction can be widely used in different problem situations, improving the effectiveness of the control method.
Patent Information
- Application Number
- CN202210553231.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-19
- Publication Date
- 2025-01-03
- Estimated Expiration
- 2042-05-19
AI Technical Summary
The existing autonomous device decision-making control method based on reinforcement learning is limited by the limited computing resources in complex problem scenarios, making it difficult to effectively train deep reinforcement learning models, and is costly and difficult to be widely used in different problem scenarios.
Using distributed reinforcement learning technology, a training system including sampling nodes, cache nodes and training nodes is established. Through distributed asynchronous maximum entropy training and concurrent acceleration model export, it realizes end-to-end autonomous device decision-making control, reduces costs and improves resource utilization .
It realizes efficient training of autonomous equipment decision-making control agents in complex problem situations, reduces the speed and cost of model training, can be widely used in different problem situations, and improves the effectiveness of control methods and resource utilization.
Smart Images
Figure CN114839879B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to an autonomous device decision control method based on distributed reinforcement learning, belonging to the technical field of autonomous device control and distributed systems. Background Art
[0002] In reality, the realization of autonomous equipment decision-making control based on reinforcement learning has the problem of large data demand. As the research on autonomous equipment decision-making control becomes more comprehensive and complicated, it is difficult to achieve results in a limited time by using only a single machine method to train autonomous equipment decision-making control agents.
[0003] In existing reinforcement learning solutions, the training of autonomous equipment decision-making and control agents is severely restricted by limited computing resources. When faced with more realistic problem scenarios with greater complexity, the deep reinforcement learning models that can be trained are limited in capabilities and often only solve single-field problems such as obstacle avoidance, path planning, and dynamic control. Summary of the invention
[0004] Purpose of the invention: In view of the problems and shortcomings in the prior art, how to use distributed conditions to train intelligent agents while maintaining high resource utilization and reducing costs is the ultimate goal of the present invention. The present invention introduces distributed deep reinforcement learning technology, and in the face of different problem scenarios, a unified method can be used to perform end-to-end autonomous device decision-making and control intelligent agent training.
[0005] The present invention proposes an autonomous device decision control method based on distributed reinforcement learning, which does not require the division of various functional modules, and can use a hierarchical reinforcement learning model to uniformly perform autonomous device decision control in complex problem scenarios. The method proposed in the present invention first establishes a training system including sampling nodes, cache nodes and training nodes, then performs distributed asynchronous maximum entropy training, and finally compiles the training results into an efficient concurrent autonomous device decision control module.
[0006] By introducing distributed deep reinforcement learning technology, we can use a unified method to train autonomous device decision-making and control agents end-to-end in the face of different problem scenarios.
[0007] Technical solution: A decision-making control method for autonomous equipment based on distributed reinforcement learning, including a training system construction step, a distributed training step, and a concurrent acceleration model export step.
[0008] In the steps of building the training system, first, a training system including sampling nodes, cache nodes and training nodes is established, an autonomous device decision-making control simulation environment is designed and implemented, and the autonomous device decision-making control simulation environment is encapsulated into a container environment. Secondly, the virtualized CPU servers and GPU servers are divided, and a virtual private network is established to ensure that all servers are in the same network segment, and then the training system is started.
[0009] In the distributed training step, the sampling node, cache node and training node are started at the same time. The sampling node continuously requests the training node to synchronize the model through the cache node agent, completes the interaction with the simulation environment, and sends the sampled data to the training node. The training node continuously receives the sampled data to the sample pool. When the sample pool capacity is sufficient, the layered maximum entropy asynchronous algorithm is used to update the reinforcement learning model parameters.
[0010] In the concurrent acceleration model export step, the training system is shut down, the training algorithm is stopped, the model parameters are downloaded from the training server, and the model parameters are packaged and compiled together with the model structure into a decision control module, and parallel optimization is started.
[0011] The establishment of the training system including sampling nodes, cache nodes and training nodes is as follows:
[0012] Step 100, containerizing and encapsulating the autonomous device decision control simulation environment;
[0013] Step 101, divide the virtualized cloud platform resources to obtain a CPU server whose main purpose is sampling and a GPU server whose main purpose is training a control model;
[0014] Step 102, establishing a virtual private network to ensure that all servers are in the same network segment;
[0015] Step 103, start the training system program, bind the sampling node and cache node to the CPU server, and bind the training node to the GPU server. If the system starts successfully, training can be performed; if the startup fails, restart the configuration from step 100.
[0016] The specific distributed asynchronous training mechanism is:
[0017] Step 200, start the training node, cache node and sampling node, and randomly initialize the control model parameters on the training node;
[0018] Step 201, the sampling node requests the training node to control model parameters, and this request is completed by the cache node proxy;
[0019] Step 202, the sampling node uses the request to control model parameters, interacts with the autonomous device decision control simulation environment, and records the complete interaction data;
[0020] Step 203, after the sampling node completes the interaction with the autonomous device decision control simulation environment, it sends the complete interaction data to the training node. This request is completed by the cache node agent, and then steps 201-203 are repeated until it is manually terminated or the preset number of cycles is reached;
[0021] Step 204: After receiving the interactive data, the training node stores it in the sample pool. If the sample capacity does not reach the preset number, wait for a period of time and repeat step 204. Otherwise, go to step 205 and execute steps 205-208 until it is manually terminated or the preset number of cycles is reached.
[0022] Step 205, sampling a batch of data from the sample pool;
[0023] Step 206, using a hierarchical maximum entropy asynchronous reinforcement learning algorithm to calculate the optimization target;
[0024] Step 207, using the back propagation algorithm to calculate the gradient and update the control model parameters;
[0025] Step 208, save the model parameters and return to step 205.
[0026] The specific method for exporting the concurrent acceleration model is as follows:
[0027] Step 300, inputting a stop command to the training system;
[0028] Step 301, download and save a series of control model parameters from the training node;
[0029] Step 302, using an automated packaging program to compile the control model parameters and the control model structure into an executable control module;
[0030] Step 303, adjust the control module to the parallel acceleration decision mode and put it into use.
[0031] The containerized encapsulation refers to the use of a function called control groups in the Linux operating system to provide an illusion of independent operation for each simulation environment, so that the autonomous device decision-making control simulation environment based on shared memory communication can run multiple instances simultaneously on the same virtual machine without interfering with each other without incurring any code modification costs.
[0032] The virtualized cloud platform can be purchased from various public cloud computing service providers or built by oneself.
[0033] The cache node exists on the same CPU server as multiple sampling nodes at the same time, and implements the synchronization model request and data sending request of the sampling node through the UnixDomainSocket communication (a cross-process communication method unique to the Linux operating system), constructing a three-layer network topology structure of training nodes, cache nodes and sampling nodes, which can effectively reduce the bandwidth cost required by the training system.
[0034] The training system includes sampling nodes, cache nodes and training nodes. Compared with other training systems, cache nodes are added. The off-track reinforcement learning algorithm allows the sampling model and the training model to have a certain degree of inconsistency. Through the cache model distribution, the sending and receiving of network data streams are executed in parallel, which greatly accelerates the operation efficiency of the entire system and greatly reduces the bandwidth requirements of the entire system, with lower costs and stronger overall performance. The off-track deep reinforcement learning algorithm is an algorithm that uses a deep neural network as a reinforcement learning model and is trained according to the off-track paradigm.
[0035] The distributed asynchronous training mechanism uses an off-track reinforcement learning algorithm based on the maximum entropy principle, allows an asynchronous workflow between sampling nodes and training nodes, and also meets the control problem's demand for smooth output; the maneuvering layer and planning layer of autonomous device control are layered, decoupling control problems at different levels and improving the effectiveness of the control method.
[0036] The concurrent acceleration model export method packages the model structure and parameters, uses compilation optimization technology to improve the operating efficiency of the overall decision-making module, and uses the parallel execution capability of multi-core processors to reduce the control decision response delay by using more computing resources when making actual autonomous device control decisions.
[0037] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the autonomous device decision control method based on distributed reinforcement learning as described above is implemented.
[0038] A computer-readable storage medium stores a computer program for executing the autonomous device decision-making control method based on distributed reinforcement learning as described above.
[0039] Beneficial effects: Compared with the prior art, the autonomous equipment decision-making and control method based on distributed reinforcement learning provided by the present invention can avoid the slow model training speed and high model training cost of the previous methods, and at the same time introduces an end-to-end hierarchical maximum entropy asynchronous algorithm, so that a decision-making and control method can be widely used in different problem situations. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Figure 1 A flow chart of the configuration of the training system according to an embodiment of the present invention;
[0041] Figure 2 This is a flow chart of the training algorithm interaction of an embodiment of the present invention;
[0042] Figure 3 A flow chart for model compilation and export according to an embodiment of the present invention. DETAILED DESCRIPTION
[0043] The present invention is further explained below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.
[0044] A method for autonomous device decision control based on distributed reinforcement learning. First, a training system including sampling nodes, cache nodes and training nodes is established, then distributed asynchronous maximum entropy training is performed, and finally the training results are compiled into an efficient concurrent autonomous device decision control module. It includes training system construction steps, distributed training steps and concurrent acceleration model export steps.
[0045] The steps for building a training system are as follows: Figure 1 As shown. In order to run multiple isolated autonomous device simulation instances on the same virtual machine at the same time, it is necessary to first encapsulate the simulation environment into a container, that is, write a container configuration file, package container dependencies, execute the container build command, and finally push it to the central container warehouse (step 100); then divide the virtualization platform resources to ensure that each virtual machine instance completely occupies all the resources of a physical machine, that is, the CPU server occupies all CPU cores and memory, and the GPU server occupies all CPU cores, memory and GPU computing cards (step 101); then establish a virtual private network so that each virtual machine can communicate with each other through a network connection represented by Ethernet (step 102); finally, write a training system configuration script, bind a proxy node and multiple sampling nodes to the CPU server, bind a training node to each GPU accelerator card of the GPU server, start the training system and check whether the startup is successful (step 103).
[0046] The distributed training steps are as follows Figure 2As shown, please note that the flowchart is not implemented completely linearly, and the sampling node and the training process have independent execution flows. Specifically, first, an instruction is issued to the training system to start the training node, cache node and sampling node in turn, and complete the initialization of the sampling node model (step 200); after the cache node is started, the model parameters are synchronized with the training node, and after the sampling node is started, the parameters are synchronized with the cache node (step 201); after the sampling node completes the model synchronization, it continuously interacts with the simulation environment container and records the complete interaction data (step 202); after the sampling node ends an interaction, it sends the recorded interaction data to the cache node, synchronizes the model from the cache node and then starts the next interaction. After the cache node accumulates a certain amount of data, it submits all the data to the training node at one time and interacts with the training node. The node performs a model synchronization (step 203); each time the training node receives interactive data, it stores it in the sample pool. If the sample pool size is insufficient, it waits for the next data transmission. If the number of samples is large enough, it proceeds to the next step (step 204); the training node randomly samples a batch of data from the sample pool (step 205); the layered maximum entropy asynchronous reinforcement learning algorithm is used to evaluate the performance of the existing model on the sampled data and give an optimization target (step 206); the back propagation algorithm is executed on the optimization target to obtain the gradient of the model parameters and update the model parameters (step 207); the updated model parameters are saved to disk and the process returns to step 205 (step 208).
[0047] The concurrent acceleration model export process is as follows Figure 3 First, input a stop command to the training system to ensure that the training algorithm stops working (step 300); download the final parameters of the autonomous device decision control model from the training server (step 301); execute the intelligent packaging program to compile the model structure and model parameters into a complete loadable dynamic link library module (step 302); adjust the decision control module to non-training mode and execute the decision control process (step 303).
[0048] Obviously, those skilled in the art should understand that the above-mentioned steps of the autonomous device decision control method based on distributed reinforcement learning of the embodiment of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present invention are not limited to any specific combination of hardware and software.
Claims
1. A decision-making control method for autonomous equipment based on distributed reinforcement learning, characterized in that: Includes training system construction steps, distributed training steps, and concurrent acceleration model export steps; In the training system construction step, firstly, a training system including sampling nodes, cache nodes and training nodes is established, an autonomous device decision control simulation environment is designed and implemented, and the autonomous device decision control simulation environment is encapsulated into a container environment. Secondly, virtualized CPU servers and GPU servers are divided, a virtual private network is established to ensure that all servers are in the same network segment, and then the training system is started; In the distributed training step, the sampling node, cache node and training node are started at the same time. The sampling node continuously requests the training node to synchronize the model through the cache node agent, completes the interaction with the simulation environment, and sends the sampled data to the training node. The training node continuously receives the sampled data to the sample pool. When the sample pool capacity is sufficient, the layered maximum entropy asynchronous algorithm is used to update the reinforcement learning model parameters. In the concurrent acceleration model export step, the training system is shut down, the training algorithm is stopped, the model parameters are downloaded from the training server, and the model parameters are packaged and compiled together with the model structure into a decision control module, and parallel optimization is started; The establishment of the training system including sampling nodes, cache nodes and training nodes is as follows: Step 100, containerizing and encapsulating the autonomous device decision control simulation environment; Step 101, divide the virtualized cloud platform resources to obtain a CPU server whose main purpose is sampling and a GPU server whose main purpose is training a control model; Step 102, establishing a virtual private network to ensure that all servers are in the same network segment; Step 103, start the training system program, bind the sampling node and the cache node to the CPU server, and bind the training node to the GPU server; if the system starts successfully, training can be performed; if the startup fails, restart the configuration from step 100; The specific distributed asynchronous training mechanism is: Step 200, start the training node, cache node and sampling node, and randomly initialize the control model parameters on the training node; Step 201, the sampling node requests the training node to control model parameters, and this request is completed by the cache node proxy; Step 202, the sampling node uses the request to control model parameters, interacts with the autonomous device decision control simulation environment, and records the complete interaction data; Step 203, after the sampling node completes the interaction with the autonomous device decision control simulation environment, it sends the complete interaction data to the training node. This request is completed by the cache node agent, and then step 201 is repeated; Step 204: After receiving the interactive data, the training node stores it in the sample pool. If the sample capacity does not reach the preset number, wait for a period of time and repeat step 204, otherwise proceed to step 205; Step 205, sampling a batch of data from the sample pool; Step 206, using a hierarchical maximum entropy asynchronous reinforcement learning algorithm to calculate the optimization target; Step 207, using the back propagation algorithm to calculate the gradient and update the control model parameters; Step 208, save the model parameters and return to step 205.
2. The autonomous device decision control method based on distributed reinforcement learning according to claim 1 is characterized in that: The specific method for exporting the concurrent acceleration model is as follows: Step 300, inputting a stop command to the training system; Step 301, download and save a series of control model parameters from the training node; Step 302, using an automated packaging program to compile the control model parameters and the control model structure into an executable control module; Step 303, adjust the control module to the parallel acceleration decision mode and put it into use.
3. The autonomous device decision control method based on distributed reinforcement learning according to claim 1 is characterized in that: The cache node exists on the same CPU server with multiple sampling nodes at the same time, and realizes the synchronization model request and the sending data request of the sampling node through the UnixDomainSocket communication agent, and constructs a three-layer network topology structure of training nodes, cache nodes and sampling nodes.
4. A computer device, characterized in that: The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the autonomous device decision-making control method based on distributed reinforcement learning as described in any one of claims 1 to 3 is implemented.
5. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for executing the autonomous device decision-making control method based on distributed reinforcement learning as described in any one of claims 1 to 3.