Distributed data processing method, device and medium of model-free reinforcement learning algorithm

Abstract

The application relates to a distributed data processing method, equipment and medium of a model-free reinforcement learning algorithm. First, roles of different task levels are encapsulated according to a task type in a reinforcement learning algorithm training process, then a communication topology structure between the roles is defined, a basic calculation unit group of a distributed communication mode is realized, the basic calculation unit group is further expanded, an aggregation and distribution mode of a preset model is defined according to selected aggregation and distribution strategies, finally, global parameter adaptive adaptation and task scheduling are performed according to a given rule until the training is completed. The single algorithm training is expanded and evolved to large-scale distributed training, from original single serial calculation, through separation of calculation components, abstraction of task role independent calculation, through task division of roles and expansion of the number of roles to achieve the purpose of rapid accumulation and exploration of samples, and parallel execution of each training task, which can greatly improve the overall sample production and training efficiency.

Description

Technical Field

[0001] The present application relates to the fields of reinforcement learning and distributed computing, and in particular to a distributed data processing method, device, and medium for scheduling model-free reinforcement learning algorithms. Background Art

[0002] The field of artificial intelligence is rapidly developing, with large-scale model technology gaining popularity. Reinforcement learning has made significant progress within this field, with examples like LLM using RLHF to optimize language models using human feedback signals, and OpenAI o1 using reinforcement technology to iteratively enhance large models. With the availability of large-scale computing power, distributed model training is gaining increasing attention from experts. While reinforcement learning has achieved impressive results, training reinforcement learning agents requires a large number of samples and, more importantly, the computing power of large-scale computing clusters. Rapidly building reinforcement learning training tasks and adapting the software and hardware of large-scale computing clusters to achieve efficient reinforcement learning training on these clusters is crucial.

[0003] There are many distributed frameworks on the market, such as Ray, PAPL, xiantian, msrl, etc. These frameworks all support distributed training of reinforcement learning algorithms. Ray is an open source distributed machine learning framework with high scalability, flexible task scheduling, multi-language support and easy integration. It uses a distributed architecture to decompose tasks into subtasks for parallel execution to improve computing efficiency. At the same time, it uses an underlying computing framework based on the actor model to efficiently utilize distributed resources. Its scheduling mechanism can dynamically schedule tasks according to task priority and resource requirements to ensure timely execution of high-priority tasks and make full use of cluster resources. However, due to the complexity of Ray's resource management and its lack of stability during use, users need to spend a lot of time and energy to learn the framework content. , which is not conducive to getting started; PARL is an open source framework for deep reinforcement learning developed by Baidu. It focuses on high efficiency and flexibility, provides simple distributed communication, flexible algorithm configuration and efficient loading of distributed data, but because distributed communication is relatively simple and highly integrated, there is no way to avoid the waste of some resources. At the same time, there are problems such as imperfect resource management, weak compatibility, and insufficient scalability; Xingtian is a highly componentized reinforcement learning library and a distributed training solution. It uses zmq for internal communication scheduling, has high throughput requirements, and can be easily and quickly started. However, the communication architecture of the distributed design is relatively complex, and the communication overhead is large. At the same time, it is difficult to freely implement or integrate an algorithm for training tasks, and the freedom and flexibility are insufficient.

[0004] In view of the problems of scalability, single model strategy and parameter quantity adaptation in the distributed training of reinforcement learning algorithms in these frameworks, how to improve them is a technical problem to be solved in this field. SUMMARY

[0005] To solve at least one of the above technical problems, an embodiment of the present application provides a distributed data processing method for a scheduling model-free reinforcement learning algorithm, comprising:

[0006] S1: encapsulating roles of different task levels according to the task type in the reinforcement learning algorithm training process;

[0007] S2: defining the communication topology structure between the roles of different task levels to realize a basic computing unit group in a distributed communication mode;

[0008] S3: expanding the basic computing unit group and presetting the aggregation and distribution mode of the model according to the selected aggregation and distribution strategy;

[0009] S4: performing global parameter self-adaptive adaptation and task scheduling according to the encapsulated roles, the basic computing unit group and the preset aggregation and distribution mode of the model until the training is completed.

[0010] Further, step S1 is specifically:

[0011] Each unit operation in the reinforcement learning algorithm training process is disassembled and encapsulated into roles of different task levels, including:

[0012] a sampler for interacting with the environment to generate samples, sending the samples to a data pool and updating parameters;

[0013] a learner for receiving the samples sent by the data pool and completing the update learning of the policy model, and sending the updated model to the sampler;

[0014] a data pool for intermediate data circulation.

[0015] Further, step S2 is specifically:

[0016] Based on the communication protocol, a data sending interface and a model receiving interface are encapsulated for the sampler; a model sending interface and a data receiving interface are encapsulated for the learner; and a data receiving interface and a data sending interface are encapsulated for the data pool;

[0017] the data sending interface of the sampler is connected with the data receiving interface of the data pool; the data sending interface of the data pool is connected with the data receiving interface of the learner; the model sending interface of the learner is connected with the model receiving interface of the sampler; to construct a basic data unit group.

[0018] Further, step S3 comprises:

[0019] S31: In each basic computing unit group, the learner is set as the master node, the sampler as the slave node, and the data pool as the data node. These three roles form a group, and the model is shared within the group;

[0020] S32: Each group is used as an expanded basic computing unit group, and a main computing unit is selected to manage the basic computing unit group; the learners in the group send the model to the main computing unit, and the model is aggregated in the main computing unit;

[0021] S33: After the aggregation is completed, the main computing unit distributes the model to each basic computing unit group, completing a model closed-loop operation, that is, a basic cycle.

[0022] Furthermore, based on the communication protocol:

[0023] In the master-slave strategy, each learner and main computing unit is encapsulated with its own model sending interface and model receiving interface; the model sending interface and model receiving interface of the main computing unit are respectively connected to the model receiving interface and model sending interface of the learner it manages.

[0024] Furthermore, based on the communication protocol:

[0025] In the neighbor strategy, the model sending interface and model receiving interface of each learner are connected to the model receiving interface and model sending interface of its neighboring learner respectively; its model is sent to its neighboring learners, and the aggregation also only aggregates the models of neighboring learners;

[0026] Alternatively, in a ring strategy, the model sending interface and model receiving interface of each learner are connected to the model receiving interface and model sending interface of the learner at the next position respectively; the model is sent to the learner at the next position, and the final communication structure presents a ring;

[0027] Or, in a dense strategy, the model sending interface and model receiving interface of each learner are connected to the model receiving interface and model sending interface of all learners except itself; its model is sent to all other learners and receives the models of all other learners;

[0028] Alternatively, in the weighted strategy, each learner evaluates the learner with the highest benefit weight compared to its current model, and its model sending interface and model receiving interface are connected to the model receiving interface and model sending interface of the learner with the highest benefit weight, so as to send its model to the learner with the highest benefit weight, and receive the model of the learner with the highest benefit weight, and aggregate according to the weight during aggregation.

[0029] Furthermore, step S4 includes:

[0030] S41: Before starting training, perform a preview of global parameters, initialize algorithm parameters, environment parameters, and scheduling parameters, and build a training task;

[0031] S42: Based on the encapsulated roles, basic computing unit groups, and the aggregation and distribution method of the preset model, a data stream communication exchange is performed to calculate basic parameters between the roles, basic computing unit groups, and the model to determine the optimal parameters;

[0032] S43: Perform training tasks according to the optimal parameters; each training task is executed in parallel according to the aggregation and distribution method of the encapsulated role, basic computing unit group, and preset model until the training is completed.

[0033] Furthermore, basic parameters include any one or more of data throughput between roles and groups, sample throughput, communication consumption, model flops, parameter quantity, theoretical maximum sample quantity, and average operation time of each unit.

[0034] In a second aspect, an embodiment of the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method described in the first aspect or any one of the implementation methods thereof is implemented.

[0035] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the method described in the first aspect or any one of the implementation methods thereof.

[0036] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0037] First, according to the task type during the reinforcement learning algorithm training process, roles of different task levels are encapsulated. Then, the communication topology between each role is defined to realize the basic computing unit group of the distributed communication mode. Then, the basic computing unit group is expanded. According to the selected aggregation and distribution strategy, the aggregation and distribution method of the model is preset. Finally, according to the established rules, global parameter adaptive adaptation and task scheduling are performed until the training is completed. It expands the training of a single algorithm to large-scale distributed training. From the original single serial calculation, by separating the computing components, it is abstracted into task roles for independent calculation. Through the division of labor and the expansion of the number of roles, the purpose of rapid accumulation and exploration of samples is achieved. Each training task is executed in parallel, which can greatly improve the overall sample production and training efficiency. In addition, based on the reinforcement learning algorithm itself, tasks can be unimpededly decomposed into functional roles. Then, different distributed training strategies are selected for different algorithms to carry out the flow of data flow and the aggregation and distribution of models. Finally, all global parameters are adaptively adjusted through the same Manager. The present invention can improve the data throughput during the overall training process. It can also greatly improve the richness of samples and greatly improve training efficiency based on the expansion of the sampler actor for reinforcement learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0039] Figure 1 A flowchart of a distributed data processing method provided in one embodiment of the present application;

[0040] Figure 2 A schematic diagram of the structure of a basic computing unit group provided in one embodiment of the present application;

[0041] Figure 3 A flow chart of the implementation of a sampler provided in one embodiment of the present application;

[0042] Figure 4 This is a flowchart of an implementation of a learner provided in one embodiment of the present application;

[0043] Figure 5 A flowchart of the implementation of the data pool provided in one embodiment of the present application;

[0044] Figure 6 A schematic diagram of the aggregation and distribution of a model provided in one embodiment of the present application;

[0045] Figure 7A schematic diagram of the structure of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0046] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0047] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0048] It will also be understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0049] As used in this specification and the appended claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0050] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0051] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0052] For ease of understanding, the technical solution of this application will be described in detail below with reference to the accompanying drawings.

[0053] Figure 1 This is a flow chart of a distributed data processing method for scheduling a model-free reinforcement learning algorithm provided in one embodiment of the present application. For ease of illustration, only the portion relevant to this embodiment is shown. The method provided in this embodiment includes:

[0054] S1: Encapsulates roles at different task levels according to the task type during the reinforcement learning algorithm training process;

[0055] Specifically, the training process of a reinforcement learning algorithm generally includes operations such as the agent interacting with the environment to generate samples, the agent using samples to update strategies, storing and operating samples, and updating strategy models. Optionally, the above operations in any reinforcement learning algorithm training process can be abstracted into roles at different task levels, and a complete reinforcement algorithm can be divided into several separate role tasks.

[0056] More specific options include Figure 2 As shown, for the above operations, each unit operation in the process is disassembled and abstracted into roles at different task levels, which may include but are not limited to:

[0057] Sampler Actor, used to interact with the environment to generate samples, send samples to the data pool, and update parameters; specifically, its number can be arbitrarily expanded and can be selected as N;

[0058] Learner: receives samples from the data pool, updates the policy model, and sends the updated model to the sampler.

[0059] Data pool Buffer, used for intermediate data flow.

[0060] More preferably, encapsulation can be implemented through programming languages, by combining functions and class codes to encapsulate various functions into roles. The specific implementation process of each role can be as follows:

[0061] like Figure 3 As shown, the specific process of the sampler Actor may include:

[0062] Analyze the game status and operating environment, enter the game if conditions permit, initialize game parameters, and reset the training scenario;

[0063] The agent uses the policy function to output actions;

[0064] Interact with the environment to obtain the next state, reward, and game status information;

[0065] The local data pool stores messages;

[0066] Record training information and monitor whether the data sending conditions are met. If so, send the data to the data pool; monitor whether the learner sends a new model. If so, update the model.

[0067] Continue iterating and returning until the training is completed and the environment is closed.

[0068] like Figure 4 As shown in the figure, the specific process of the learner may include:

[0069] Analyze the game status and operating environment, enter the game if conditions permit, initialize game parameters, and reset the training scenario;

[0070] Determine whether the data pool sends samples. If so, store the information in the local data pool or directly prepare for strategy learning.

[0071] Determine whether the policy update learning conditions are met. If so, the policy is updated and learned; otherwise, the model and information are saved.

[0072] Monitor whether the model sending conditions are met, and if so, send the model to the sampler;

[0073] Continue iterating and returning until the training is completed and the environment is closed.

[0074] like Figure 5 As shown in the figure, the specific steps of the data pool Buffer may include:

[0075] Analyze the game status and operating environment. If the conditions are met, monitor whether the sampler has sent data. If so, store the sample data in the data pool. Determine whether the data sending conditions are met. If so, send the sample data to the learner.

[0076] Continue iterating and returning until the training is completed and the environment is closed.

[0077] For example, most reinforcement learning problems can be converted into MDP problems; therefore, we can choose to use the Markov decision process MDP modeling in reinforcement learning as an example. The modeling process design uses the Bellman equation to recursively update and evaluate the value of each state, thereby gradually approaching the optimal strategy. Based on this foundation, the characteristics of value iteration and policy iteration are used to further optimize the decision-making process. TD (temporal difference algorithm) is an important value function evaluation method in reinforcement learning. It combines the advantages of Monte Carlo and dynamic programming. At the same time, TD can be used for both on-policy and off-policy learning, meeting the requirements of unified abstraction for current reinforcement learning algorithms. The specific steps include:

[0078] Step 1: Initialize the state value function V(s);

[0079] Step 2: Initialize the state S for each episode;

[0080] Step 3: When state S is not a terminal state, select action a according to the current strategy;

[0081] Step 4: Execute action a, observe reward r and next state s';

[0082] Step 5: The agent performs learning updates, including but not limited to updating the TD target, updating the TD error calculation, updating the value function, etc.

[0083] Step 6: Update the state S = S'. If S is the terminal state, the episode ends. Otherwise, go to step 3 and repeat the loop.

[0084] By adopting the distributed data processing method of the present invention, the above process steps can be encapsulated into roles:

[0085] The sampler actor role is mainly for the task encapsulation of steps 3 and 4. The agent selects actions according to the current strategy in state S and observes the reward and the next state. This process is both interacting with the environment and obtaining this sample (s, r, s', a);

[0086] The learner role is mainly responsible for encapsulating the tasks in step 5 and performing learning updates on the agent.

[0087] The data pool Buffer role is an intermediate role. When the learner and the sampler Actor calculate separately, the samples produced by the sampler Actor will be transferred to the data pool Buffer, and then sent to the learner for learning calculation.

[0088] The above three roles constitute the unified encapsulation of the reinforcement learning algorithm. Although different reinforcement learning algorithms have certain degrees of differences, they will not deviate from the abstract scope of these three tasks, that is, complete the overall encapsulation of a complete reinforcement algorithm.

[0089] S2: Based on the different task levels of roles, define the communication topology between each other and realize the basic computing unit group of distributed communication mode;

[0090] Specifically, the training process of the reinforcement learning algorithm includes operations such as the interaction between the agent and the environment to generate samples, the agent using samples to update strategies, storing and operating samples, and updating strategy models. Figure 2As shown in the figure, the data communication between the roles mainly includes sample transfer and model transmission. Sample transfer refers to the data pool Buffer receiving samples from the sampler Actor and sending samples to the learner Learner, while model transmission means that the strategy model updated by the learner Learner needs to be consistent with the intelligent model of the sampler Actor.

[0091] More specifically, based on the communication protocol, the communication requirements of each role are adaptively modified and encapsulated on the native MPI communication interface, creating separate communication semantics for each task role. Different communication topology modes are then designed based on different communication topologies to provide ready-made communication modes to meet the different distributed training functions of the reinforcement algorithm. This includes:

[0092] Based on the communication protocol, encapsulate the data sending interface and model receiving interface for the sampler Actor; encapsulate the model sending interface and data receiving interface for the learner Learner; encapsulate the data receiving interface and data sending interface for the data pool Buffer;

[0093] The data sending interface of the sampler is connected to the data receiving interface of the data pool; the data sending interface of the data pool is connected to the data receiving interface of the learner; the model sending interface of the learner is connected to the model receiving interface of the sampler.

[0094] After the above encapsulation, each role can complete its communication tasks. The sampler's communication tasks are: 1) sending data to the buffer for processing, and 2) receiving the model from the learner. The learner's communication tasks are: 1) sending the model to the actor, and 2) receiving data from the buffer. The data pool's communication tasks are: 1) receiving data from the actor, and 2) sending data to the learner. These three constitute the most basic computing unit group. Expanding within this group, based on actors, allows for up to N sample production roles, significantly improving production efficiency.

[0095] S3: Expands the basic computing unit group and presets the aggregation and distribution methods of the model based on the selected aggregation and distribution strategies;

[0096] Specifically, you can choose to use reduce and scatter strategies, such as master-slave strategy, neighbor strategy, ring strategy, dense strategy, weight strategy, etc. to expand and manage the basic computing unit group, and preset the aggregation and distribution mode of the model.

[0097] More specifically, taking the master-slave strategy as an example, step S3 includes:

[0098] S31: In each basic computing unit group, the learner is set as the master node, the sampler actor is set as the slave node, and the data pool buffer is set as the data node. These three roles form a group, and the model is shared within the group;

[0099] S32: Each group is used as an expanded basic computing unit group, and a main computing unit Rooter is selected to manage the basic computing unit group; the learner in the group sends the model to the main computing unit Rooter, and the model is aggregated in the main computing unit Rooter;

[0100] S33: After the aggregation is completed, the main computing unit Rooter distributes the model to each basic computing unit group, completing a model closed-loop operation, that is, a basic cycle.

[0101] In this embodiment, a model aggregation and distribution method based on a master-slave strategy is provided. A master computing unit is selected, and each basic computing unit group is managed by its own learner. The master computing unit Rooter manages the learners in multiple basic computing unit groups. In terms of specific tasks, the learner Leaner, in addition to being responsible for communicating with the sampler Actor and the data pool Buffer, also needs to communicate with the master computing unit Rooter. Therefore, the definition of the communication topology between each other also includes: Figure 6 As shown,

[0102] Based on the communication protocol, the model sending interface and model receiving interface are encapsulated for the learner Learner and the main computing unit Rooter; the model sending interface and model receiving interface of the main computing unit Rooter are respectively connected to the model receiving interface and model sending interface of the learner Learner it manages.

[0103] In this embodiment, a model aggregation and distribution method set according to the master-slave strategy is given, and a preferred embodiment of the communication topology is set. Under this topology, the learner Leaner can send the current model to the main computing unit Rooter, and at the same time receive the model sent by the main computing unit Rooter. The main computing unit Rooter can receive models sent by multiple learner Leaners and send models to multiple learner Leaners at the same time. Specifically, all model sending and receiving can optionally use synchronous interface communication. In addition, other strategies such as neighboring, ring, dense, weighted, etc. can also be used for model aggregation and distribution. Example:

[0104] Neighborhood strategy: The model sending interface and model receiving interface of each learner are connected to the model receiving interface and model sending interface of its adjacent learner respectively; its model is sent to its adjacent learner, and only the models of adjacent learners are aggregated.

[0105] Ring strategy: The model sending interface and model receiving interface of each learner are connected to the model receiving interface and model sending interface of the learner at the next position respectively; its model is sent to the learner at the next position, and the final communication structure presents a ring.

[0106] Dense strategy: The model sending interface and model receiving interface of each learner are connected to the model receiving interface and model sending interface of all learners except itself; its model is sent to all other learners and receives the models of all other learners.

[0107] Weight strategy: Each learner evaluates the learner with the highest weight compared to its current model. Its model sending interface and model receiving interface are connected to the model receiving interface and model sending interface of the learner with the highest weight, so as to send its model to the learner with the highest weight, and receive the model of the learner with the highest weight. During aggregation, the main computing unit aggregates according to the weight.

[0108] S4: Perform global parameter adaptive adaptation and task scheduling based on the encapsulated roles, basic computing unit groups, and the aggregation and distribution methods of the preset models until training is completed.

[0109] Specifically, there are many parameters involved in reinforcement learning training, including agent hyperparameters, model parameters, algorithm parameters, and environment parameters. Therefore, in order to adapt and better adjust and optimize parameters, this method provides a global manager to adaptively adjust and optimize all hyperparameters; including:

[0110] S41: Before starting training, perform a preview of global parameters, initialize algorithm parameters, environment parameters, and scheduling parameters, and build a training task. Specifically, options include automatic scheduling of adaptation parameters, automatic allocation of the number of computing cores and the amount of communication data, etc.

[0111] S42: Based on the aggregation and distribution method of the encapsulated roles, basic computing unit groups, and preset models, a data stream communication exchange is performed to calculate the basic parameters between the roles, basic computing unit groups, and models to determine the optimal parameters; specifically, optional but not limited to data throughput, sample throughput, communication consumption, model flops, parameter quantity, theoretical maximum sample quantity, average operation time of each unit, etc. between roles and groups, and unified analysis is performed by the global manager to calculate the optimal parameters and make global parameter adjustments.

[0112] S43: Perform training tasks according to the optimal parameters; each training task is executed in parallel according to the aggregation and distribution method of the encapsulated role, basic computing unit group, and preset model until the training is completed.

[0113] Specifically, each training task process executes in parallel. The sampler Actor task process is responsible for sample production, the learner task process performs learning and policy updates, and the data pool Buffer task process manages and transmits samples. Within a single communication group, all tasks are managed uniformly by its own learner. The learner of each computing unit group is aggregated and distributed by the Rooter.

[0114] In summary, the present invention provides a distributed data processing method for scheduling model-free reinforcement learning algorithms. First, according to the task type in the reinforcement learning algorithm training process, roles of different task levels are encapsulated. Then, the communication topology between the roles is defined to implement the basic computing unit group of the distributed communication mode. Then, the basic computing unit group is expanded. According to the selected aggregation and distribution strategy, the aggregation and distribution method of the model is preset. Finally, global parameter adaptive adaptation and task scheduling are performed according to established rules until the training is completed. It expands the training of a single algorithm to large-scale distributed training. From the original single serial calculation, by separating the computing components, it is abstracted into task roles for independent calculation. Through the division of labor and the expansion of the number of roles, the purpose of rapid accumulation and exploration of samples is achieved. The training tasks are executed in parallel, which can significantly improve the overall sample production and training efficiency. In addition, based on the reinforcement learning algorithm itself, tasks can be unimpededly decomposed into functional roles. Then, different distributed training strategies are selected for different algorithms to carry out the flow of data flow and the aggregation and distribution of models. Finally, all global parameters are adaptively adjusted through the same manager. The present invention can improve the data throughput during the overall training process, and can also greatly improve the richness of samples and greatly improve training efficiency based on the expansion of the sampler Actor for reinforcement learning.

[0115] Taking the training of navigation models such as robot dogs and drones as an example, according to the distributed data processing method of this application, the following options can be selected:

[0116] 1. Encapsulate the PPO algorithm training process into roles, define the communication topology between roles, build a basic computing unit group, then expand the basic computing unit group, preset the model aggregation and distribution method, and complete the preliminary preparations;

[0117] 2. Initialize algorithm parameters, environment parameters, and scheduling parameters, and build training tasks;

[0118] 3. The Manager performs a rehearsal, completing a round of learning and data rotation, calculating the model's flops, sample throughput, and communication time. The Manager analyzes the rehearsal data and calculates the optimal parameters.

[0119] 4. After parameter adjustment, training officially begins. Each training task process executes in parallel. The Actor task process is responsible for sample production, the Learner task process performs learning and policy updates, and the Buffer task process manages and transmits samples. Within this communication group, the Learner process manages all tasks. The Rooter aggregates and distributes models for each compute unit group's Learner.

[0120] 5. After the training is completed, analyze the training results.

[0121] like Figure 7 FIG. 1 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application, including: at least one processor 70 ( Figure 7 Only one is shown), a memory 71 and a computer program 72 stored in the memory 71 and executable on at least one processor 70, the processor 70 executes the computer program 72 to implement the above Figure 1 Steps in a method embodiment.

[0122] The electronic device 7 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The electronic device 7 may include but is not limited to a processor 70 and a memory 71. Those skilled in the art will appreciate that Figure 7 It is only an example of the electronic device 7 and does not constitute a limitation on the electronic device 7. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0123] The processor 70 may be a central processing unit (CPU), or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0124] In some embodiments, the memory 71 may be an internal storage unit of the electronic device 7, such as a hard disk or memory of the electronic device 7. In other embodiments, the memory 71 may also be an external storage device of the electronic device 7, such as a plug-in hard disk equipped on the electronic device 7, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Furthermore, the memory 71 may include both an internal storage unit of the electronic device 7 and an external storage device. The memory 71 is used to store an operating system, application programs, a boot loader, data, and other programs, such as the program code of a computer program. The memory 71 may also be used to temporarily store data that has been output or is about to be output.

[0125] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0126] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the process in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program, when executed by the processor, can implement the steps of the above-mentioned various method embodiments. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include at least: any entity or device capable of carrying the computer program code to an electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electric carrier signal, a telecommunication signal and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk.

[0127] A computer-readable storage medium provided in an embodiment of the present application has the same beneficial effects as the above method.

[0128] An embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it can implement the steps in the above-mentioned various method embodiments.

[0129] A computer program product provided in an embodiment of the present application has the same beneficial effects as the above method.

[0130] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0131] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0132] In the embodiments provided in this application, it should be understood that the disclosed devices / equipment and methods can be implemented in other ways. For example, the device / equipment embodiments described above are merely schematic. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0133] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0134] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A distributed data processing method for scheduling model-free reinforcement learning algorithms, characterized in that: The method comprises: S1: Encapsulates roles at different task levels according to the task type during the reinforcement learning algorithm training process; S2: Based on the different task levels of roles, define the communication topology between each other and realize the basic computing unit group of distributed communication mode; S3: Expands the basic computing unit group and presets the aggregation and distribution methods of the model based on the selected aggregation and distribution strategies; S4: Perform global parameter adaptive adaptation and task scheduling based on the encapsulated roles, basic computing unit groups, and the aggregation and distribution methods of the preset models until training is completed; Based on communication protocol: In the neighbor strategy, the model sending interface and model receiving interface of each learner are connected to the model receiving interface and model sending interface of its neighboring learner respectively; its model is sent to its neighboring learners, and the aggregation also only aggregates the models of neighboring learners; Alternatively, in a ring strategy, the model sending interface and model receiving interface of each learner are connected to the model receiving interface and model sending interface of the learner at the next position respectively; the model is sent to the learner at the next position, and the final communication structure presents a ring; Or, in a dense strategy, the model sending interface and model receiving interface of each learner are connected to the model receiving interface and model sending interface of all learners except itself; its model is sent to all other learners and receives the models of all other learners; Alternatively, in a weighted strategy, each learner evaluates the learner with the highest weight of benefit compared to its current model, and its model sending interface and model receiving interface are connected to the model receiving interface and model sending interface of the learner with the highest weight of benefit, so as to send its model to the learner with the highest weight of benefit, and receive the model of the learner with the highest weight of benefit, and aggregate according to the weight; Step S4 includes: S41: Before starting training, perform a preview of global parameters, initialize algorithm parameters, environment parameters, and scheduling parameters, and build a training task; S42: Based on the encapsulated roles, basic computing unit groups, and the aggregation and distribution method of the preset model, a data stream communication exchange is performed to calculate basic parameters between the roles, basic computing unit groups, and the model to determine the optimal parameters; S43: Perform training tasks based on the optimal parameters; each training task is executed in parallel according to the aggregation and distribution method of the encapsulated roles, basic computing unit groups, and preset models until the training is completed; Basic parameters include data throughput between roles and groups, sample throughput, communication consumption, model flops, number of parameters, theoretical maximum sample size, and any one or more of the average operation time of each unit.

2. The distributed data processing method according to claim 1, wherein: Step S1 is specifically as follows: Each unit operation in the reinforcement learning algorithm training process is disassembled and encapsulated into roles at different task levels, including: Sampler, used to interact with the environment to generate samples, send samples to the data pool, and update parameters; The learner receives samples from the data pool, updates the policy model, and sends the updated model to the sampler. Data pool, used for intermediate data flow.

3. The distributed data processing method according to claim 1, wherein: Step S2 is specifically as follows: Based on the communication protocol, encapsulate the data sending interface and model receiving interface for the sampler; encapsulate the model sending interface and data receiving interface for the learner; encapsulate the data receiving interface and data sending interface for the data pool; The data sending interface of the sampler is connected to the data receiving interface of the data pool; the data sending interface of the data pool is connected to the data receiving interface of the learner; the model sending interface of the learner is connected to the model receiving interface of the sampler; so as to construct a basic data unit group.

4. The distributed data processing method according to claim 1, wherein: Step S3 includes: S31: In each basic computing unit group, the learner is set as the master node, the sampler as the slave node, and the data pool as the data node. These three roles form a group, and the model is shared within the group. S32: Each group is used as an expanded basic computing unit group, and a main computing unit is selected to manage the basic computing unit group; the learners in the group send the model to the main computing unit, and the model is aggregated in the main computing unit; S33: After the aggregation is completed, the main computing unit distributes the model to each basic computing unit group, completing a model closed-loop operation, that is, a basic cycle.

5. The distributed data processing method according to claim 4, characterized in that: Based on communication protocol: In the master-slave strategy, each learner and main computing unit is encapsulated with its own model sending interface and model receiving interface; the model sending interface and model receiving interface of the main computing unit are respectively connected to the model receiving interface and model sending interface of the learner it manages.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 5 is implemented.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images