Data processing method and device for distributed reinforcement learning system
By deploying serverless computing functions in distributed reinforcement learning systems, the agent acquires and releases CPU and GPU resources on demand, solving the problem of resource waste in existing systems and improving training efficiency.
Patent Information
- Application Number
- CN202411695167.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-25
- Publication Date
- 2025-03-18
AI Technical Summary
When existing reinforcement learning training systems perform interaction with the environment and update the policy network, they lead to idle waste of computing resources such as CPU and GPU, and cannot effectively utilize resources, affecting training efficiency.
By deploying the data processing process of the distributed reinforcement learning system as a serverless computing function, the agent acquires CPU and GPU resources in the form of a serverless computing process, uses it as needed and recycles resources in a timely manner to avoid resource idle waste.
It realizes efficient utilization of CPU and GPU resources, avoids resource waste, and improves the efficiency and speed of reinforcement learning training.
Smart Images
Figure CN119645632A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present invention relate to the field of cloud computing technology, and in particular to a data processing method and device for a distributed reinforcement learning system. Background Art
[0002] The training process of reinforcement learning usually includes the following key steps: 1. Define the environment and state space; 2. Design the reward function; 3. Build the agent; 4. Interact with the environment and collect experience data; 5. Update the agent's policy network; 6. Repeat steps 4 and 5 until the training converges.
[0003] The fourth step of the above reinforcement learning training process (interacting with the environment and collecting experience data) mainly relies on the central processing unit (CPU) resources, while the fifth step (updating the agent's policy network) mainly relies on the graphics processing unit (GPU) computing resources. In the existing training system, when executing step 4, the GPU resources are idle and not used; when executing step 5, the CPU resources are idle and not used. There are also some reinforcement learning algorithms that require steps 4 and 5 to be executed alternately. The existing reinforcement learning training system inevitably leads to idle computing resources and waste of resources. Summary of the invention
[0004] The embodiments of the present application provide a data processing method and device for a distributed reinforcement learning system, which continuously interacts with the environment, learns and optimizes its own decision-making strategy based on empirical data, and ultimately completes the target task of data processing in a desired manner.
[0005] In a first aspect, an embodiment of the present application provides a data processing method of a distributed reinforcement learning system, comprising:
[0006] Any agent in the working node obtains CPU resources in the form of a serverless computing process; and generates experience data through the CPU resources;
[0007] After any agent in the working node writes the experience data into the first shared memory, the agent is stopped and the CPU resources are released;
[0008] The data transmission device in the working node sends the experience data in the first shared memory to the data buffer of the master node; the master node is used to update the strategy network parameters according to the experience data sent by each working node; the strategy network parameters are used to indicate the work of any working node in the next round.
[0009] This application deploys the data processing work process of the distributed reinforcement learning system as a serverless computing function, uses CPU computing resources and GPU resources on demand and recycles them in a timely manner, avoiding idle waste of computing resources.
[0010] Optionally, the process manager in the working node obtains the policy network parameters from the master node and stores them in the second shared memory;
[0011] The process manager in the working node starts the next round of agents, and any agent obtains the policy network parameters through the second shared memory.
[0012] Optionally, the process manager in the working node obtains the number of agents for the next round from the master node;
[0013] The agent that starts the next round includes:
[0014] The process manager in the working node starts the agents corresponding to the number of agents for the next round.
[0015] By dynamically adjusting the number of agents in each round, the training efficiency of reinforcement learning is greatly improved and the time required for reinforcement learning is reduced. By designing and running agents with lightweight serverless computing functions, efficient computing expansion and contraction are achieved, greatly reducing the time overhead of dynamically adjusting the number of working processes each time.
[0016] Optionally, the working node sets each agent to have its own exclusive CPU resources.
[0017] Optionally, stopping the agent comprises:
[0018] After the working node takes a snapshot of the agent's context and stores it in the hard disk, it stops the agent;
[0019] The agent that starts the next round includes:
[0020] The working node starts the next round of agents based on the snapshot.
[0021] This enables faster allocation and release of CPU computing resources for intelligent agents.
[0022] In a second aspect, an embodiment of the present application provides a data processing method for a distributed reinforcement learning system, for any round of reinforcement learning, including:
[0023] Any learner in the master node obtains GPU resources in the form of a serverless computing process;
[0024] Any learner in the master node obtains experience data from the data buffer of the master node; the experience data is stored in the data buffer through the data transmitter of any working node; and the experience data is generated by any agent in any working node after obtaining CPU resources in the form of a serverless computing process; after the experience data is stored in the first shared memory in the working node, the agent stops working and releases the CPU resources;
[0025] Any learner of the master node updates the policy network parameters through the GPU resources according to the experience data sent by each working node, and releases the GPU resources after the update.
[0026] This application deploys the data processing work process of the distributed reinforcement learning system as a serverless computing function, uses CPU computing resources and GPU resources on demand and recycles them in a timely manner, avoiding idle computing resources.
[0027] Optionally, the master node determines the number of agents for the next round;
[0028] The master node sends the policy network parameters and the number of agents for the next round to each working node by broadcasting.
[0029] By dynamically adjusting the number of agents in each round, the training efficiency of reinforcement learning is greatly improved and the time required for reinforcement learning is reduced. By designing and running agents with lightweight serverless computing functions, efficient computing expansion and contraction are achieved, greatly reducing the time overhead of dynamically adjusting the number of working processes each time.
[0030] Optionally, the first acquisition module is used to acquire CPU resources in the form of a serverless computing process; and generate experience data through the CPU resources;
[0031] The first acquisition module is also used to stop the agent and release the CPU resources after writing the experience data into the first shared memory;
[0032] The first acquisition module is also used to send the experience data in the first shared memory to the data buffer of the master node;
[0033] The first processing module is used to update the policy network parameters according to the experience data sent by each working node; the policy network parameters are used to indicate the work of any working node in the next round.
[0034] In a third aspect, the distributed reinforcement learning system includes a first acquisition module and a first processing module, and for any round of reinforcement learning, includes:
[0035] The first acquisition module is used to acquire CPU resources in the form of a serverless computing process; and generate experience data through the CPU resources;
[0036] The first acquisition module is also used to stop the agent and release the CPU resources after writing the experience data into the first shared memory;
[0037] The first acquisition module is also used to send the experience data in the first shared memory to the data buffer of the master node;
[0038] The first processing module is used to update the policy network parameters according to the experience data sent by each working node; the policy network parameters are used to indicate the work of any working node in the next round.
[0039] Optionally, the first processing module is further used for:
[0040] Acquire the policy network parameters from the master node and store them in the second shared memory;
[0041] The next round of agents is started, and any agent obtains the policy network parameters through the second shared memory.
[0042] Optionally, the first processing module is further used to obtain the number of agents for the next round from the master node;
[0043] Start the number of agents that matches the next round.
[0044] Optionally, the first processing module is further used to set each agent to have its own exclusive CPU resources.
[0045] Optionally, the first processing module is further used to stop the agent after taking a snapshot of the agent's context and storing it in a hard disk;
[0046] Based on the snapshot, the next round of agents is started.
[0047] In a fourth aspect, an embodiment of the present application provides a data processing device of a distributed reinforcement learning system, wherein the distributed reinforcement learning system includes a second acquisition module and a second processing module, and for any round of reinforcement learning, includes:
[0048] The second acquisition module is used to acquire GPU resources in the form of a serverless computing process;
[0049] The second acquisition module is further used to acquire experience data from the data buffer of the master node; the experience data is stored in the data buffer through the data transmitter of any working node; and the experience data is generated after any agent in any working node acquires CPU resources in the form of a serverless computing process; after the experience data is stored in the first shared memory in the working node, the agent stops working and releases the CPU resources;
[0050] The second processing module is used to update the policy network parameters through the GPU resources according to the experience data sent by each working node, and release the GPU resources after the update.
[0051] Optionally, the second processing module is further used for:
[0052] Determine the number of agents for the next round;
[0053] The policy network parameters and the number of agents for the next round are sent to each working node by broadcasting.
[0054] In a fifth aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes any of the task allocation methods described in the first aspect above.
[0055] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program executable by a computer device, wherein when the program runs on the computer device, the computer device executes any of the task allocation methods described in the first aspect above.
[0056] In a seventh aspect, an embodiment of the present application provides a computer program product, which includes a computer program / instruction, and when the computer program / instruction is executed by a processor, it implements any task allocation method described in the first aspect above. BRIEF DESCRIPTION OF THE DRAWINGS
[0057] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0058] Figure 1 A schematic diagram of the structure of a data processing system of a distributed reinforcement learning system provided in an embodiment of the present application;
[0059] Figure 2A schematic diagram of a data processing method for a distributed reinforcement learning system provided in an embodiment of the present application;
[0060] Figure 3 A schematic diagram of the structure of a data processing system of a distributed reinforcement learning system provided in an embodiment of the present application;
[0061] Figure 4 A schematic diagram of a data processing device for a distributed reinforcement learning system provided in an embodiment of the present application;
[0062] Figure 5 A schematic diagram of a data processing device for a distributed reinforcement learning system provided in an embodiment of the present application;
[0063] Figure 6 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0064] In order to make the purpose, technical solution and beneficial effects of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.
[0065] An agent is an agent that can perceive the environment and take actions to achieve specific goals. It can be software, hardware or a system with autonomy, adaptability and interaction. Agents perceive changes in the environment (such as through sensors or data input), make judgments and decisions based on their own learned knowledge and algorithms, and then perform actions to affect the environment or achieve predetermined goals. Agents are widely used in the field of artificial intelligence and are commonly seen in automated systems, robots, virtual assistants and game characters. The core of agents is the ability to learn autonomously and continuously evolve to better complete tasks and adapt to complex environments.
[0066] Non-Uniform Memory Access (NUMA) is a computer memory design for multiple processors where memory access time depends on the memory location of the processor. Under NUMA, a processor accesses its own local memory faster than non-local memory (memory location to another processor or memory shared between processors).
[0067] CRIU (Checkpoint / Restore In Userspace) is a software tool running on the Linux operating system. Its function is to implement the Checkpoint / Restore function in the user space.
[0068] Serverless computing is a cloud computing execution model that allows developers to focus on writing and deploying code without having to worry about the management and maintenance of the underlying infrastructure: 1. The serverless computing platform automatically adjusts computing resources according to actual user needs, without the need to manually configure and manage servers. 2. Users only need to pay for the computing resources actually consumed, rather than purchasing a fixed number of servers or capacity in advance. 3. Fast deployment: Developers can quickly write and deploy functions without configuring and managing the underlying infrastructure. Serverless computing provides a high degree of scalability, flexibility, and cost-effectiveness, allowing developers to focus more on business logic rather than infrastructure management.
[0069] Reinforcement Learning: Reinforcement Learning (RL) is an important branch of machine learning that focuses on how to enable an agent to learn optimal strategies through interaction with the environment to maximize long-term rewards. Reinforcement learning has a wide range of applications in many fields, such as game artificial intelligence (such as chess, Go, Atari games), robot control, autonomous driving, recommendation systems, etc. It provides a general learning framework that enables agents to continuously improve their decision-making capabilities through interaction with the environment.
[0070] Distributed reinforcement learning training: Distributed reinforcement learning training is a method to accelerate training and improve model performance by distributing the training process of reinforcement learning algorithms to multiple computing nodes. It takes advantage of distributed computing to process large-scale training data and explore the environment in parallel, thereby speeding up learning and obtaining better strategies.
[0071] First of all, we need to know that the goal of reinforcement learning training is to enable the agent to learn an optimal strategy so that the agent can obtain the maximum cumulative reward and achieve the predetermined goal in the process of interacting with the environment.
[0072] Then, the training process of reinforcement learning usually includes the following key steps:
[0073] 1. Define the environment and state space
[0074] Clearly define the environment in which the agent is located, including the state of the environment, action space, transition probability, reward function and other elements. The state space describes the set of states that the environment is in at a certain moment. The state can be discrete or continuous.
[0075] 2. Design reward function
[0076] The reward function defines the feedback that the agent gets from the environment after performing an action. The design of the reward function needs to take into account the final training goal and guide the agent to learn and optimize the strategy in the desired direction.
[0077] 3. Building an Agent
[0078] Define a learnable agent, usually a neural network, also called a policy network, to approximate the optimal policy. The agent receives the state of the environment as input and outputs the action that should be taken in that state.
[0079] 4. Interact with the environment and collect empirical data
[0080] The agent interacts with the environment at each time step, selects an action based on the current state, and obtains the reward and the state of the next moment. The four-tuple experience data (s, a, r, s') collected over a period of time is saved for subsequent training. Among them, s represents the current state of the environment; a represents the action that the agent will take in the environment state s; r represents the reward obtained by the agent after taking action a in the environment state s; s' represents the new state that the environment enters after the agent takes action a in the environment state s.
[0081] 5. Update the agent’s policy network
[0082] Based on a batch of four-tuple experience data (s, a, r, s'), the current agent's policy network updates the neural network parameters based on the loss function and optimization algorithm (such as gradient descent).
[0083] 6. Repeat steps 4 and 5 until training converges
[0084] Repeat the process of interacting with the environment to collect data and update strategies until a certain number of iterations is reached or the performance indicators meet the requirements.
[0085] In short, the core training process of reinforcement learning is that the intelligent agent learns and optimizes its decision-making strategy based on reward signals through continuous interaction with the environment, and finally completes the target task in the desired way. This requires reasonable definition of state-action space, design of reward function, construction of strategy network, and balance between exploration and utilization. Training is a process of trial and error learning and gradual progress.
[0086] like Figure 1As shown, it is a structural diagram of a data processing system 100 of a distributed reinforcement learning system provided by the present application, including a master node 101 and multiple working nodes 102. Among them, the master node includes a data buffer 103 and multiple learners 104, and each working node 102 includes multiple agents 105 and a policy network 106; any agent 105 obtains CPU resources in the form of a serverless computing process; any learner 104 obtains GPU resources in the form of a serverless computing process.
[0087] like Figure 2 As shown, it is a flow chart of a data processing method of a distributed reinforcement learning system provided by the present application, which specifically includes the following steps for any round of reinforcement learning:
[0088] Step S201: Any agent in a working node obtains CPU resources in the form of a serverless computing process; and generates experience data through the CPU resources.
[0089] Specifically, the agent in the working node interacts directly with the environment at each time step. The agent obtains environmental information through the CPU resources obtained. The environmental information includes the state of the environment, action space, transition probability, reward function and other information. The agent selects an action based on the current state and obtains the reward and the state at the next moment. The four-tuple experience data (s, a, r, s') collected over a period of time is saved for subsequent training. Among them, s represents the current state of the environment; a represents the action that the agent will take in the environment state s; r represents the reward obtained by the agent after taking action a in the environment state s; s' represents the new state that the environment enters after the agent takes action a in the environment state s.
[0090] Step S202: After any agent in the working node writes the experience data into the first shared memory, the agent is stopped and the CPU resources are released.
[0091] Specifically, in each working node, there is a first shared memory and multiple agents. The shared memory is used to store the experience data generated by the agents in all working nodes. After each agent finishes interacting with its environment, the CPU resources occupied by the agent will be released.
[0092] Step S203: the data transmitter in the working node sends the experience data in the first shared memory to the data buffer of the master node.
[0093] Specifically, after the intelligent agent in each working node completes the interaction with the environment, the data transmitter in the working node will send the experience data in the first shared memory in the working node to the data buffer in the master node.
[0094] Step S204: any learner in the master node obtains GPU resources in the form of a serverless computing process.
[0095] Specifically, there is at least one learner in the master node, and the learner is used to read the experience data sent by the working node from the data buffer.
[0096] Step S205: any learner in the master node obtains experience data from the data buffer of the master node.
[0097] Step S206, the master node updates the policy network parameters according to the experience data sent by each working node, and releases the GPU resources after the update; the policy network parameters are used to indicate the work of any working node in the next round.
[0098] Specifically, the master node reads the experience data sent by the working node from the data buffer, uses GPU resources to calculate the experience data, updates the policy network parameters, and releases GPU resources after the policy network parameters are updated. The master node broadcasts the updated policy network parameters to each working node, and the working node performs the next round of work according to the updated policy network parameters.
[0099] This application deploys the data processing work process of the distributed reinforcement learning system as a serverless computing function, uses CPU computing resources and GPU resources on demand and recycles them in a timely manner, avoiding idle waste of computing resources.
[0100] In the above step S203, after the data transmitter in the working node sends the environmental information in the first shared memory to the master node, it also includes: the process manager in the working node obtains the strategy network parameters from the master node and stores them in the second shared memory; the process manager in the working node starts the next round of intelligent agents, and any intelligent agent obtains the strategy network parameters through the second shared memory.
[0101] Specifically, a process manager is deployed on each working node. The process manager of each working node is responsible for receiving the latest policy network parameters from the master node and storing them in the second shared memory for the agent to read. At the same time, the process manager deployed on each working node is also used to start (invoke / restore) the function of the next round of agents.
[0102] In this application, in order to allocate and release the CPU computing resources of the agent more quickly, we use CRIU (Checkpoint / Restore In Userspace) to implement lightweight startup and snapshot of the agent.
[0103] Specifically, when the agent is stopped and its CPU computing resources are released, the working node stores the state data of the agent's context in the form of a snapshot (checkpoint) to the hard disk; when the agent needs to be started in the next round, the working node directly reads the state data of the context in the hard disk based on the snapshot (checkpoint) to quickly start and obtain CPU resources for the agent.
[0104] In some embodiments, before starting the next round of intelligent agents, it also includes: the process manager in the working node obtains the number of intelligent agents for the next round from the master node; starting the next round of intelligent agents includes: the process manager in the working node starts the intelligent agents that meet the number of intelligent agents for the next round.
[0105] Specifically, the process manager in the master node dynamically sets the number of agents required for the next round of parallel work according to the training progress of the policy network of the working node. The master node sends the policy network parameters and the number of agents for the next round to each working node by broadcasting. The process manager in the working node starts (invoke / restore) the corresponding number of agents according to the number of agents required for the next round of parallel work received, and uses these agents to collect a new round of experience data.
[0106] By dynamically adjusting the number of agents in each round, the training efficiency of reinforcement learning is greatly improved and the time required for reinforcement learning is reduced. By designing and running agents with lightweight serverless computing functions, efficient computing expansion and contraction are achieved, greatly reducing the time overhead of dynamically adjusting the number of working processes each time.
[0107] In a possible embodiment, the working node sets each agent to have its own exclusive CPU resources. CPU isolation between processes can be performed by changing the NUMA (Non-Uniform Memory Access) policy, that is, a specific CPU core is allocated to a specific process to improve performance and reduce resource competition. Among them, according to the resource configuration amount preset by the user, the NUMA policy can be modified when starting each actor process to meet the user's configured resource allocation.
[0108] like Figure 3As shown, it is an architecture diagram of a data processing system of a distributed reinforcement learning system provided by the present application, and the whole system includes a master node (Learner) and several worker nodes (Woker). In each worker node, a process manager (process manager), multiple agents (agents), a first shared memory (shm actor data), a policy network (agent policy network), and a first data transfer (data transfer client) are deployed. In the master node, a data buffer (shm actor data buffer), multiple learners (learner process), a second data transfer (data transfer server), a workload monitor (workload monitor), a workload dispatcher (workload dispatcher), and a GPU service (GPU service) are deployed.
[0109] The following is a detailed description of the interaction process between the working node and the master node in any round of reinforcement learning. A working node contains multiple agents, and each agent obtains CPU computer resources in the form of an actor process. The agent interacts directly with the environment at each time step. The agent selects an action based on the current state to obtain the reward and the state at the next moment. The four-tuple experience data (s, a, r, s') collected over a period of time is stored in the first shared memory (shm actor data). After each agent interacts with its environment, the CPU resources occupied by the agent will be released. The first data transfer client in the working node sends the experience data in the first shared memory (shm actor data) to the second data transfer server of the master node. The second data transfer server sends the experience data to the data buffer (shm actor data buffer). After any learner process in the master node obtains GPU resources in the form of a serverless computing process, it obtains the experience data from the data buffer (shm actor data buffer) of the master node for calculation and updates the policy network parameters. After the policy network parameters are updated, the GPU resources are released. Among them, the workload monitor is responsible for monitoring the execution time of each round of training tasks; the workload dispatcher is responsible for dispatching trigger instructions to the working nodes in each round, that is, the master node broadcasts the updated policy network parameters to each working node through the workload dispatcher, and deploys a process manager (Process Manager) on each working node. The process manager of each working node is responsible for receiving the latest policy network parameters from the master node and storing them in the second shared memory for the agent to read. At the same time, the process manager deployed on each working node is also used to start (invoke / restore) the function of the next round of agents.
[0110] In the above process, the master node can dynamically adjust the number of actor processes in each round, and dynamically allocate these actor processes to different worker nodes according to resource idleness. Since we can store actorprocesses in the form of image files on disk, users only need to set the maximum number of agents required, and register and save these agents on disk in the first round. In this way, when it is detected that the number of agents needs to be increased, the worker node can simply restore the actor processes from the image file. Since the image file has been registered in advance, this is no different from a normal startup. As a result, we can achieve rapid expansion without the need to re-initialize the actors and incur additional overhead.
[0111] In the above embodiment, in order to facilitate algorithm engineers to use distributed training more easily, we designed a simple programming interface centered on the data buffer (shm actor data buffer). Specifically, the data buffer, as a medium for storing experience data, connects the two functional modules of policy network update and experience data collection: policy network update needs to read data from the data buffer, and experience data collection needs to write the collected data to the data buffer. Under the programming paradigm centered on the data buffer, algorithm engineers only need to specifically define the interaction between policy network update and experience data collection and the data buffer, without having to worry about the implementation and maintenance of the underlying network data transmission.
[0112] like Figure 4 As shown, a data processing device 400 of a distributed reinforcement learning system provided by the present application includes a first acquisition module 401 and a first processing module 402, and for any round of reinforcement learning, includes:
[0113] The first acquisition module 401 is used to acquire CPU resources in the form of a serverless computing process; and generate experience data through the CPU resources;
[0114] The first acquisition module 401 is also used to stop the agent and release the CPU resources after writing the experience data into the first shared memory;
[0115] The first acquisition module 401 is also used to send the experience data in the first shared memory to the data buffer of the master node;
[0116] The first processing module 402 is used to update the policy network parameters according to the experience data sent by each working node; the policy network parameters are used to indicate the work of any working node in the next round.
[0117] This application deploys the data processing work process of the distributed reinforcement learning system as a serverless computing function, uses CPU computing resources and GPU resources on demand and recycles them in a timely manner, avoiding idle waste of computing resources.
[0118] Optionally, the first processing module 402 is further configured to:
[0119] Acquire the policy network parameters from the master node and store them in the second shared memory;
[0120] The next round of agents is started, and any agent obtains the policy network parameters through the second shared memory.
[0121] Optionally, the first processing module 402 is further used to obtain the number of agents for the next round from the master node;
[0122] Start the number of agents that matches the next round.
[0123] Optionally, the first processing module 402 is further used to set each agent to have its own exclusive CPU resources.
[0124] Optionally, the first processing module 402 is further configured to stop the agent after taking a snapshot of the agent's context and storing it in a hard disk;
[0125] Based on the snapshot, the next round of agents is started.
[0126] like Figure 5 As shown, a data processing device 500 of a distributed reinforcement learning system is provided in an embodiment of the present application. The distributed reinforcement learning system includes a second acquisition module 501 and a second processing module 502. For any round of reinforcement learning, it includes:
[0127] The second acquisition module 501 is used to acquire GPU resources in the form of a serverless computing process;
[0128] The second acquisition module 501 is also used to acquire experience data from the data buffer of the master node; the experience data is stored in the data buffer through the data transmitter of any working node; and the experience data is generated after any agent in any working node acquires CPU resources in the form of a serverless computing process; after the experience data is stored in the first shared memory in the working node, the agent stops working and releases the CPU resources;
[0129] The second processing module 502 is used to update the policy network parameters through the GPU resources according to the experience data sent by each working node, and release the GPU resources after the update.
[0130] This application deploys the data processing work process of the distributed reinforcement learning system as a serverless computing function, uses CPU computing resources and GPU resources on demand and recycles them in a timely manner, avoiding idle waste of computing resources.
[0131] Optionally, the second processing module 502 is further configured to:
[0132] Determine the number of agents for the next round;
[0133] The policy network parameters and the number of agents for the next round are sent to each working node by broadcasting.
[0134] Based on the same technical concept, the embodiment of the present application provides a computer device, which can be Figure 6 The terminal device 601 and / or the server 602 shown in FIG. Figure 6 As shown, it includes at least one processor 601 and a memory 602 connected to the at least one processor. The specific connection medium between the processor 601 and the memory 602 is not limited in the embodiment of the present application. Figure 6 For example, the processor 601 and the memory 602 are connected via a bus. The bus can be divided into an address bus, a data bus, a control bus, etc.
[0135] In an embodiment of the present application, the memory 602 stores instructions that can be executed by at least one processor 601. The at least one processor 601 can execute the steps of the data processing method of the above-mentioned distributed reinforcement learning system by executing the instructions stored in the memory 602.
[0136] Among them, the processor 601 is the control center of the computer device, and various interfaces and lines can be used to connect various parts of the computer device. By running or executing instructions stored in the memory 602 and calling data stored in the memory 602, the computing resources of the above-mentioned distributed reinforcement learning system can be released and recovered in time to avoid waste of computing resources. Optionally, the processor 601 may include one or more processing units, and the processor 601 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 601. In some embodiments, the processor 601 and the memory 602 may be implemented on the same chip, and in some embodiments, they may also be implemented separately on separate chips.
[0137] Processor 601 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware processor for execution, or can be executed by a combination of hardware and software modules in the processor.
[0138] The memory 602 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 602 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, etc. The memory 602 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer device, but is not limited thereto. The memory 602 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.
[0139] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program that can be executed by a computer device. When the program runs on the computer device, the computer device executes the steps of the data processing method of the above-mentioned distributed reinforcement learning system.
[0140] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0141] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0142] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0143] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0144] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.
Claims
1. A data processing method for a distributed reinforcement learning system, characterized in that: The distributed reinforcement learning system includes a master node and a working node, and for any round of reinforcement learning, includes: Any agent in the working node obtains CPU resources in the form of a serverless computing process; and generates experience data through the CPU resources; After any agent in the working node writes the experience data into the first shared memory, the agent is stopped and the CPU resources are released; The data transmission device in the working node sends the experience data in the first shared memory to the data buffer of the master node; the master node is used to update the strategy network parameters according to the experience data sent by each working node; the strategy network parameters are used to indicate the work of any working node in the next round.
2. The method according to claim 1, characterized in that After the data transmitter in the working node sends the environment information in the first shared memory to the master node, the method further includes: The process manager in the working node obtains the policy network parameters from the master node and stores them in the second shared memory; The process manager in the working node starts the next round of agents, and any agent obtains the policy network parameters through the second shared memory.
3. The method according to claim 2, characterized in that Before starting the next round of agents, the process further includes: The process manager in the working node obtains the number of agents for the next round from the master node; The agent that starts the next round includes: The process manager in the working node starts the agents corresponding to the number of agents for the next round.
4. The method according to any one of claims 1 to 3, characterized in that Also includes: The working node sets each agent to have its own exclusive CPU resources.
5. The method according to claim 4, characterized in that The stopping agent comprises: After the working node takes a snapshot of the agent's context and stores it in the hard disk, it stops the agent; The agent that starts the next round includes: The working node starts the next round of agents based on the snapshot.
6. A data processing method for a distributed reinforcement learning system, characterized in that: The distributed reinforcement learning system includes a master node and a working node, and for any round of reinforcement learning, includes: Any learner in the master node obtains GPU resources in the form of a serverless computing process; Any learner in the master node obtains experience data from the data buffer of the master node; the experience data is stored in the data buffer through the data transmitter of any working node; and the experience data is generated by any agent in any working node after obtaining CPU resources in the form of a serverless computing process; after the experience data is stored in the first shared memory in the working node, the agent stops working and releases the CPU resources; Any learner of the master node updates the policy network parameters through the GPU resources according to the experience data sent by each working node, and releases the GPU resources after the update.
7. The method according to claim 6, characterized in that After releasing the GPU resources after updating, the method further includes: The master node determines the number of agents for the next round; The master node sends the policy network parameters and the number of agents for the next round to each working node by broadcasting.
8. A data processing device for a distributed reinforcement learning system, characterized in that: The distributed reinforcement learning system includes a first acquisition module and a first processing module, and for any round of reinforcement learning, includes: The first acquisition module is used to acquire CPU resources in the form of a serverless computing process; and generate experience data through the CPU resources; The first acquisition module is also used to stop the agent and release the CPU resources after writing the experience data into the first shared memory; The first acquisition module is also used to send the experience data in the first shared memory to the data buffer of the master node; The first processing module is used to update the policy network parameters according to the experience data sent by each working node; the policy network parameters are used to indicate the work of any working node in the next round.
9. A data processing device for a distributed reinforcement learning system, characterized in that: The distributed reinforcement learning system includes a second acquisition module and a second processing module, and for any round of reinforcement learning, includes: The second acquisition module is used to acquire GPU resources in the form of a serverless computing process; The second acquisition module is further used to acquire experience data from the data buffer of the master node; the experience data is stored in the data buffer through the data transmitter of any working node; and the experience data is generated after any agent in any working node acquires CPU resources in the form of a serverless computing process; after the experience data is stored in the first shared memory in the working node, the agent stops working and releases the CPU resources; The second processing module is used to update the policy network parameters through the GPU resources according to the experience data sent by each working node, and release the GPU resources after the update.
10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of any method described in claims 1 to 5 or 6 to 7 are implemented.
11. A computer-readable storage medium, characterized in that: It stores a computer program executable by a computer device. When the program is run on the computer device, the computer device executes the steps of any method described in claims 1 to 5 or 6 to 7.
12. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps of the method described in any one of claims 1 to 5 or 6 to 7 are implemented.