A deep reinforcement learning computing platform based on a Ray parallel distributed framework

Abstract

The application relates to the field of reinforcement learning, and particularly discloses a deep reinforcement learning computing platform based on a Ray parallel distributed framework; the application combines and extends parallel and distributed methods, takes the Ray parallel distributed framework as an underlying framework, integrates open source frameworks related to reinforcement learning, is compatible with various robot simulation environments, so that the platform can not only fully utilize cluster hardware resources, but also improve the speed of reinforcement learning training data acquisition. In addition, the platform utilizes a docker container technology, greatly improves the deployment speed of the platform, and through experiments, it can be proved that the platform is a computing platform with availability, scalability and stability.

Description

Technical Field

[0001] This invention belongs to the field of reinforcement learning technology, specifically relating to a deep reinforcement learning computing platform based on the Ray parallel distributed framework. Background Technology

[0002] Deep learning primarily trains multi-layered neural networks to achieve desired learning outcomes. It combines low-level features to form more abstract high-level representations of attribute categories or features, discovering distributed feature representations of data. Reinforcement learning is an important learning method in machine learning and is considered an independent branch of Markov decision processes and dynamic optimization methods. Reinforcement learning involves agents learning through trial and error. With the development of artificial intelligence, reinforcement learning is no longer limited to discrete environments with small action and state spaces. Deep reinforcement learning, in particular, faces more complex and realistic continuous environments, with robotics being one of its main application scenarios. Currently, most methods used in robotics are based on control theory, requiring the pre-establishment of models and the collection of large amounts of interaction data between the robot and its environment to train the control strategy model. This is costly and requires extensive robot interactions to obtain large amounts of sampled data. However, using real robots for large-scale data sampling also presents problems, such as slow sampling speed, high data loss, high cost, and even potential dangers, such as with quadcopter robots. To address the difficulties in data collection, the University of Southern California developed an advanced robot simulator—Gazebo—which helps save costs, shorten time, and accelerate data collection. To facilitate environment simulation and robot training, many excellent projects exist, such as Gym and Robotschoo L under OpenAI. Satisfactory training results can only be obtained through sufficient interaction and training with the robot environment. However, due to the limitations of the single-instruction sequential operation mode, training time has become a bottleneck for reinforcement learning applications in the field of robotics.

[0003] To address these issues, a deep reinforcement learning computing platform based on the Ray parallel distributed architecture can be designed, combining and extending parallel and distributed methods to improve the reinforcement learning training speed of robots. Summary of the Invention

[0004] The purpose of this invention is to provide a deep reinforcement learning computing platform based on the Ray parallel distributed framework to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] A deep reinforcement learning computing platform based on the Ray parallel distributed framework, specifically comprising:

[0007] Application layer one, which consists of a lightweight desktop based on Docker virtual LXDE and VNC control components;

[0008] Ray Parallel Distributed Framework: The Ray parallel distributed framework is used to reduce distributed iterative jobs from the task level to the function level;

[0009] The simulation framework integrates the Torcs racing game for autonomous driving testing tasks, the VizDoom FPS game framework for image recognition and scenario interaction testing tasks, the robot simulation software gazebo, and the Bullet simulation environment.

[0010] The virtual container layer uses Docker virtualization container technology to containerize the platform and improve its flexibility.

[0011] Preferably, the application layer structure includes a browser, VNC remote desktop service, Nginx reverse proxy service, Docker virtual desktop and cluster. After the service is started, users can access the virtual LXDE lightweight desktop using a browser, submit training model tasks, sample simulated environment data, and conveniently view training results through the virtual Ubuntu operating system desktop. The Nginx reverse proxy service can effectively prevent the cluster within the local area network from being attacked.

[0012] Preferably, the Ray parallel distributed framework includes:

[0013] Application layer two, which includes multiple nodes, each node is equipped with a local scheduling module, an object storage module and a drive / actor / worker module;

[0014] The system layer includes a global control module, a global scheduling module, and a global storage list module.

[0015] Preferably, the Ray parallel distributed framework can distribute tasks asynchronously and in parallel, and can share all storage resources. The nodes of the application layer two are interconnected, and the system layer is interconnected with the nodes of the application layer two.

[0016] Preferably, the deployment and startup components of the simulation framework are modified according to the parallel distributed and containerized deployment, as well as the robot simulation framework adapts to online sampling and training content.

[0017] Preferably, the simulation framework has built-in a variety of open-source reinforcement learning algorithm toolkits, including Gym, TensorFlow, PyTorch, and Tune and RL lib provided by the Ray development team.

[0018] Compared with the prior art, the beneficial effects of the present invention are:

[0019] This invention combines and extends parallel and distributed methods, using the Ray parallel distributed framework as the underlying framework and integrating relevant open-source frameworks for reinforcement learning. It is compatible with various robot simulation environments, enabling the platform to fully utilize cluster hardware resources and improve the speed of reinforcement learning training data acquisition. Furthermore, the platform utilizes Docker container technology, significantly increasing deployment speed. Experiments demonstrate that this platform is a computing platform that combines availability, scalability, and stability. Attached Figure Description

[0020] Figure 1 This is a diagram of the overall system architecture of the present invention;

[0021] Figure 2 This is an application layer architecture diagram of the present invention;

[0022] Figure 3 This is a diagram of the Ray parallel distributed architecture of the present invention;

[0023] Figure 4 This is a comparison table of Docker and virtual machine architectures in this invention;

[0024] Figure 5 This is a hardware and system configuration specification table for the experimental platform of the present invention;

[0025] Figure 6 This is a comparison chart of the streaming data experiments of the present invention;

[0026] Figure 7 This is a comparison table of streaming data experiments for the present invention;

[0027] Figure 8 This is an experimental diagram illustrating the scalability of the RRLP platform of the present invention.

[0028] Figure 9 This is the RRLP platform extensibility table of the present invention. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0030] Example:

[0031] Please see Figure 1 - Figure 9 As shown, a deep reinforcement learning computing platform based on the Ray parallel distributed framework specifically includes:

[0032] Application layer one consists of a lightweight desktop based on Docker virtual LXDE and VNC control components, allowing users to perform desktop operations conveniently and intuitively, and view the simulation and training effects of the robot in near real-time;

[0033] Ray Parallel Distributed Framework: The Ray parallel distributed framework is used to reduce distributed iterative jobs from the task level to the function level;

[0034] The simulation framework integrates the Torcs racing game for autonomous driving testing tasks, the VizDoom FPS game framework for image recognition and scenario interaction testing tasks, the robot simulation software gazebo, and the Bullet simulation environment.

[0035] The virtual container layer uses Docker virtualization container technology to containerize the platform and improve its flexibility.

[0036] The application layer structure includes a browser, VNC remote desktop service, Nginx reverse proxy service, Docker virtual desktop, and cluster. After the service starts, users can access the virtual LXDE lightweight desktop through the browser, submit training model tasks, sample simulated environment data, and conveniently view training results through the virtual Ubuntu operating system desktop. The Nginx reverse proxy service can effectively prevent the cluster within the local area network from being attacked.

[0037] The Ray parallel distributed framework includes:

[0038] Application layer two includes multiple nodes, each node being equipped with a local scheduling module, an object storage module, and a drive / actor / worker module;

[0039] The system layer includes a global control module, a global scheduling module, and a global storage list module.

[0040] Spark's distributed general-purpose architecture employs a master-slave architecture, utilizing computer memory for large-scale data processing, which effectively supports the training of machine learning models. While Spark can be applied to conventional machine learning models, its task-level job iteration method struggles to effectively handle reinforcement learning tasks that require frequent parameter exchanges. The asynchronous parallel framework Ray, suitable for deep reinforcement learning, reduces distributed iterative jobs from the task level to the function level, effectively addressing Spark's lack of support for reinforcement learning. Therefore, the RRLP platform proposed in this invention uses Ray as its main framework, perfectly resolving the frequent data collection and redistribution during reinforcement learning model training. Ray not only distributes tasks asynchronously and in parallel but also shares all storage resources, thus achieving superior performance more suitable for parallel distributed reinforcement learning computation.

[0041] This invention's simulation architecture integrates the Torcs racing game for autonomous driving testing, the VizDoom FPS game framework for image recognition and contextual interaction testing, the commonly used robot simulation software Gazebo, and the Bullet simulation environment. Since the simulation framework system is designed as a single machine, modifications were made to the deployment and startup components of each simulation framework to facilitate parallel, distributed, and containerized deployment, making the robot simulation framework more adaptable to online sampling and training. Furthermore, to facilitate the training of deep reinforcement learning models, the platform includes various open-source reinforcement learning algorithm toolkits, including Gym, TensorFlow, PyTorch, and Tune and RLB provided by the Ray development team.

[0042] Most existing reinforcement learning training models and robot simulation frameworks are developed based on the Linux system. However, users use different operating systems with varying versions. Therefore, Docker virtualization container technology was introduced to enable containerized deployment of the platform, thereby improving its flexibility and effectively solving the compatibility problem between multiple systems. Docker's unique virtualization method has significant advantages over traditional virtual machines. Developers can quickly deliver and deploy products without the need for additional virtualization management programs, achieving efficient utilization of computer resources, easy migration and scaling, and rapid version management.

[0043] Experiments were conducted on a deep reinforcement learning computing platform based on the RAY parallel distributed row architecture and the Spark platform, focusing on performance in both streaming data processing and node scalability. The results were then used to analyze the platforms. Figure 5 This document describes the hardware and software system configuration for the CPU and GPU experimental platform. All experiments in this invention are based on this configuration.

[0044] Streaming data processing performance is an important metric for evaluating machine learning or reinforcement learning, effectively reflecting the data computation performance and data interaction speed between nodes in a system. Therefore, streaming data processing was used to test the performance of the RRLP platform. Additionally, the classic Spark distributed framework was chosen as a comparative experiment. The basic data for the experiment consisted of 20 English journal articles and 20 Chinese journal articles, with each article having a memory size of approximately 29Mb. The experiment involved counting the number of characters, words, and lines in the articles. The number of nodes selected ranged from 1 to 10, with each node corresponding to 12 CPU cores.

[0045] pass Figure 6 The comparison chart of streaming data experiments shows that the RRLP platform takes less time to process data than the Spark distributed computing platform, and the RRLP platform performs better when the number of running nodes is small. Figure 7 The data processing time for the same number of running nodes is presented. It can be seen that the RRLP platform saves between 81 and 234 seconds compared to Spark for streaming data processing, representing a saving rate between 2.22% and 8.49%. In conclusion, the RRLP platform outperforms Spark in streaming data processing, and its performance steadily improves with the number of nodes, demonstrating a certain degree of stability.

[0046] In conducting distributed experiments on the RRLP platform, in addition to hardware configuration, scalability is another very important performance indicator of the distributed platform. Therefore, this experiment uses different numbers of nodes to determine whether the RRLP computing platform has good scalability.

[0047] The simulation environment used in the experiment was the HumanoidBulletEnv-0 simulation environment in PyBullet. The policy selected was the reinforcement learning policy algorithm: A3C (asynchronous advantage actor-critic). Hyperparameters were selected by hyperparameter tuning. The termination condition of the experiment was that the optimal reward reached 1000. A GPU cluster was used, with 2-12 nodes in the cluster. Figure 8 This is a line graph showing the performance of the RRLP platform at different numbers of running nodes. Figure 9 This refers to the number of iterations required to reach the stopping condition. Analysis shows that for the same training task, the RRLP platform steadily reduces training time as the number of nodes increases, indicating that the increase in the number of nodes does not excessively increase inter-node communication overhead. In conclusion, the RRLP platform possesses a certain degree of stability and scalability.

[0048] This invention addresses the limitations of traditional computing platforms in meeting the demands of current deep reinforcement learning development by proposing an asynchronous parallel training platform—the Deep Reinforcement Learning Computing Platform (RRLP) based on the RAY parallel distributed architecture. This platform uses the Ray parallel distributed framework as its underlying structure, integrates relevant open-source frameworks for reinforcement learning, and is compatible with various robot simulation environments. This allows the platform to fully utilize cluster hardware resources and improve the speed of reinforcement learning training data acquisition. Furthermore, the platform utilizes Docker container technology, significantly enhancing deployment speed. Experiments demonstrate that the RRLP platform is a usable, scalable, and stable computing platform.

[0049] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0050] The accompanying drawings of the embodiments disclosed in this invention only involve structures relevant to the embodiments disclosed in this invention. Other structures can be referred to with common designs. Unless otherwise specified, the same embodiment and different embodiments of this invention can be combined with each other.

Claims

1. A deep reinforcement learning computing platform based on the Ray parallel distributed framework, characterized in that, Specifically, it includes: Application layer one, which consists of a lightweight desktop based on Docker virtual LXDE and VNC control components; Ray Parallel Distributed Framework: The Ray parallel distributed framework is used to reduce distributed iterative jobs from the task level to the function level; The simulation framework integrates the Torcs racing game for autonomous driving testing tasks, the VizDoom FPS game framework for image recognition and scenario interaction testing tasks, the Gazebo robot simulation software, and the Bullet simulation environment. The virtual container layer uses Docker virtualization container technology to containerize the platform and improve its flexibility.

2. The deep reinforcement learning computing platform based on the Ray parallel distributed framework according to claim 1, characterized in that: The application layer structure includes a browser, VNC remote desktop service, Nginx reverse proxy service, Docker virtual desktop and cluster. After the service starts, users can access the virtual LXDE lightweight desktop through the browser, submit training model tasks, sample simulated environment data, and conveniently view training results through the virtual Ubuntu operating system desktop. The Nginx reverse proxy service can effectively prevent the cluster within the local area network from being attacked.

3. The deep reinforcement learning computing platform based on the Ray parallel distributed framework according to claim 1, characterized in that: The Ray parallel distributed framework includes: Application layer two includes multiple nodes, each node is equipped with a local scheduling module, an object storage module and a drive / actor / worker module; The system layer includes a global control module, a global scheduling module, and a global storage list module.

4. The deep reinforcement learning computing platform based on the Ray parallel distributed framework according to claim 3, characterized in that: The Ray parallel distributed framework can distribute tasks asynchronously and in parallel, and can share all storage resources. The nodes of the application layer 2 are interconnected, and the system layer is interconnected with the nodes of the application layer 2.

5. A deep reinforcement learning computing platform based on the Ray parallel distributed framework according to claim 4, characterized in that: The deployment and startup components of the simulation framework are modified to adapt to online sampling and training content based on parallel distributed and containerized deployments and the robot simulation framework.

6. A deep reinforcement learning computing platform based on the Ray parallel distributed framework according to claim 5, characterized in that: The simulation framework includes a variety of open-source reinforcement learning algorithm toolkits, including Gym, TensorFlow, PyTorch, and Tune and RLlib provided by the Ray development team.

Images