Reinforcement learning algorithm-oriented data processing method, system and equipment and medium

Abstract

The invention relates to a reinforcement learning algorithm-oriented data processing method, system and device and a medium, and the method comprises the steps: carrying out the configuration processing of a target parameter and a target reinforcement algorithm, and can conveniently and rapidly adjust the target parameter, so as to achieve the optimization training effect. Time sequence difference is used as a basic process of interaction of an intelligent agent and a data environment, so that modeling of a unified training normal form is solved, an interface function, an experience playback pool and various auxiliary operations are matched, a reinforcement learning algorithm and a reinforcement learning data environment can be quickly and conveniently accessed, meanwhile, the algorithm is efficiently modified, debugged and optimized, and the accuracy of the reinforcement learning algorithm and the reinforcement learning data environment is improved. And the logic burden caused by algorithm implementation is reduced.

Description

Technical Field

[0001] The present application relates to the field of reinforcement learning and intelligent agent game technology, and in particular to a data processing method, system, device and medium for reinforcement learning algorithms. Background Art

[0002] At present, reinforcement learning is widely used in the fields of robotics, transportation, logistics, etc. With the rapid development of artificial intelligence technology, the ability of neural networks to approximate function values ​​has been further explored. Setting up an experience replay pool similar to the hippocampus and updating the strategy network and the target network off-track has made a qualitative breakthrough in reinforcement learning. From the unstable and non-convergent performance of reinforcement learning algorithms in the 10s of the last century, to now surpassing the level of human professional players in decision-making tasks such as Atari, StarCraft, and Dota2.

[0003] Reinforcement learning algorithms need to set different training parameters and handle different learning update boundaries when facing different application environments or scenarios, so that the same reinforcement algorithm will be processed differently when trained in different environments. In the field of intelligent agent reinforcement learning, there are many reinforcement learning algorithm components or frameworks, such as RLlib, Baselines, cleanrl, corl and other reinforcement learning frameworks. These frameworks all support reinforcement learning algorithm training, evaluation and other functions. Among them, the RLlib reinforcement learning algorithm library implements many reinforcement learning algorithms based on the pytorch and tensorflow deep learning frameworks. During the training phase, RLlib can directly call reinforcement algorithms through the command line to quickly implement training tasks, but the disadvantage is that the overall training scheduling system is very complex, with many sub-processes, large engineering volume, and difficulty in getting started. There are barriers to freely modifying the algorithm and training process. Stable Baselines, cleanrl, corl and other frameworks integrate a series of reinforcement learning algorithms. The technical solutions for training are basically to provide training instances of the corresponding data environment of the algorithm respectively. In the training phase, the training implementation of the algorithm is directly integrated as a module and then imported for direct use, that is, they all provide a specific training task for the algorithm and integrate the training implementation of the algorithm. For example, the DQN reinforcement learning algorithm has its own separate training process implementation, and the same is true for DDPG, PPO, Qmix, TD3 and other algorithms. When switching from one algorithm training task to another, you are faced with a new training logic, in which the codes such as agent abstraction, interface semantic definition, hyperparameter positioning, and algorithm strategy logic will have different degrees of differences. This difference is particularly evident in different environmental applications and different algorithm types (such as off-policy algorithms and on-policy algorithms), which ultimately leads to higher workload and time costs as the complexity of the algorithm task increases. Summary of the invention

[0004] Based on this, the purpose of this application is to provide a method, system, device and medium for data processing for reinforcement learning algorithms to solve the technical problems mentioned in the above background technology.

[0005] In a first aspect, the present application provides a data processing method for a reinforcement learning algorithm. The method is applied in a data environment including an intelligent agent; the method includes:

[0006] Determine the interface function associated with the reinforcement algorithm to achieve the adaptation of the target reinforcement algorithm and the data environment;

[0007] Configuring the target parameters and the target reinforcement algorithm to obtain a training task related to the configuration file;

[0008] Taking temporal difference as the basic process of interaction between the agent and the data environment, and combining the experience replay pool and multiple auxiliary operations to obtain a training paradigm;

[0009] The data processing of the training task is completed through the training paradigm.

[0010] In one embodiment, the reinforcement algorithm includes a single-agent algorithm and a multi-agent algorithm; the interface function corresponds to the multi-layer interface abstracted by the reinforcement algorithm, including a model layer model, an algorithm layer embryo and an agent layer; wherein: the model layer model is defined using a neural network, and implements the model definition part and the reasoning part; the algorithm layer embryo points to the reinforcement learning algorithm to update the learning part, and the corresponding interface function includes at least an update function update and a prediction function execute; the agent layer represents an entity that interacts with the data environment through code logic, and the corresponding interface function includes at least a learning function learn and a policy function policy.

[0011] In one embodiment, the adaptation of the target reinforcement algorithm and the data environment is achieved, including: determining the environment interface corresponding to the data environment; the functions of the environment interface at least include environment initialization init, action interaction step, scene rendering render, and environment reset reset; ensuring the consistency of the environment interface semantics when the intelligent agent corresponding to the target reinforcement algorithm interacts with the data environment.

[0012] In one embodiment, the target parameters include at least initialization parameters, model parameters, and environment parameters; the target parameters and the target reinforcement algorithm are configured to obtain a training task related to the configuration file, including: implementing the configuration processing of the target parameters and the target reinforcement algorithm, and decoupling the code logic to obtain the configuration file; the configuration file is processed by a registration module to obtain the training task; the registration module includes at least an optimizer, a learning rate scheduler, and a strategy scheduler.

[0013] In one embodiment, multiple auxiliary operations at least include adding hooks and training information to the basic process of temporal difference; the training paradigm obtained by combining the experience replay pool and multiple auxiliary operations includes: when the data environment enters the game state, calling the hook to obtain the strategy factor and the learning factor; the intelligent agent uses the strategy factor to call the policy function policy, and determines the trajectory data after interacting with the data environment; the trajectory data includes the next state, reward and game state; calling the hook before learning the target reinforcement algorithm to perform pre-learning preparations; the intelligent agent calls the learning function learn according to the trajectory data and the learning factor, and updates the strategy of the target reinforcement algorithm according to the experience replay pool; calling the hook after learning the target reinforcement algorithm to perform post-learning preparations and store training information; the training information includes the number of learning steps, reward value, and number of experience samples.

[0014] In one embodiment, the strategy of the target reinforcement algorithm is updated according to the experience replay pool, including: determining the type of experience replay pool corresponding to the target reinforcement algorithm, and performing initialization operations; determining the current experience bar element of the agent interacting with the data environment, and structuring the current experience bar element through a data structure; when the experience replay pool is a single-step experience pool, learning and updating the agent until the game state ends.

[0015] In one embodiment, the above method also includes: when the experience replay pool is a multi-step experience pool and the current step number in the game is not the first step, judging whether the internal serial number of the sample is the game number of this game; if so, adding the elements of this experience bar to the various queues of the multi-step experience pool respectively, and the number of steps in the game is increased by one; if not, performing structured processing on the various queues of the multi-step experience pool, and adding them as experience samples until the game status ends.

[0016] In the second aspect, the present application provides a data processing system for reinforcement learning algorithms. It includes:

[0017] The interface function module is used to determine the interface function associated with the reinforcement algorithm and realize the adaptation of the target reinforcement algorithm and the data environment;

[0018] A configuration processing module, used for configuring the target parameters and the target reinforcement algorithm to obtain a training task related to the configuration file;

[0019] The training paradigm module is used to use the temporal difference as the basic process of interaction between the intelligent agent and the data environment, and to obtain the training paradigm by combining the experience replay pool and a variety of auxiliary operations; the data processing of the training task is completed through the training paradigm.

[0020] In a third aspect, the present application further provides a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the above data processing steps for the reinforcement learning algorithm when executing the computer program.

[0021] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above steps of data processing for the reinforcement learning algorithm are implemented.

[0022] The above-mentioned data processing method, system, computer device and readable storage medium for reinforcement learning algorithm are aimed at the situation that multiple types of algorithm training cannot be combined in reinforcement learning training. In order to make the switching and use of training tasks simple and convenient, the target parameters and the target reinforcement algorithm are configured, and the target parameters can be adjusted conveniently and quickly to achieve the effect of optimizing training. By using time series difference as the basic process of interaction between the intelligent agent and the data environment, the modeling of the unified training paradigm is solved. With the interface function, experience replay pool and various auxiliary operations, the reinforcement learning algorithm and the reinforcement learning data environment can be quickly and conveniently accessed, and the algorithm can be efficiently modified, debugged and optimized, reducing the logical burden brought by the algorithm implementation. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A framework diagram of a data processing system for a reinforcement learning algorithm in one embodiment;

[0024] Figure 2 is a flow chart of a data processing method for a reinforcement learning algorithm in one embodiment;

[0025] Figure 3 A schematic diagram of a process of accessing an algorithm training task in one embodiment;

[0026] Figure 4 A pseudo code diagram of a timing difference process in one embodiment;

[0027] Figure 5 A schematic diagram of a process of calling a reinforcement learning algorithm within a system in one embodiment;

[0028] Figure 6 A flowchart of a training paradigm in one embodiment;

[0029] Figure 7 A schematic diagram of a process of performing storage determination in an experience replay pool in one embodiment;

[0030] Figure 8 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0032] The data processing method for reinforcement learning algorithm provided in the embodiment of the present application can be executed by the computer device associated with the intelligent agent. The operation of selecting the computer device is in the instantiation process of generating the training task. The selection method can be directly determined from the initialization parameters of the intelligent agent, or it can be selected from the configuration file, and the method is relatively free. Among them, the computer device can be a terminal or a server, and the corresponding computing platform can be a CPU or a GPU. The terminal includes but is not limited to various laptops, smart phones, tablet computers, drone equipment, smart car-mounted equipment and portable wearable devices. The server can be implemented with an independent server or a server cluster composed of multiple servers. It can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0033] The framework structure design of the data processing system for the reinforcement learning algorithm provided in the embodiment of the present application is as follows: Figure 1 As shown in the figure, the general training and data processing of learning algorithms can be enhanced through the design of the framework structure. The design concept of the system framework structure is based on modularity, scalability, single responsibility and simplicity. The entire framework structure can be divided into three aspects: training body, configuration plug-in and function plug-in.

[0034] Among them, the training subject is divided into the agent module, the environment module, and the training platform. The agent module includes the agent design and the agent function plug-in. The agent design is divided into the model layer model, the algorithm layer embryo and the agent layer. The agent function plug-in has the functions of the agent hook, the agent buffer, the agent strategy, etc. The agent hook is also the agent hook, which is a functional supplement to the agent algorithm process. Specifically, it transfers the special algorithm operations to the agent hook for operation; the agent buffer is also the experience replay pool, which is used to store the data structure obtained by the interaction between the agent and the environment; the agent strategy is mainly the strategy adopted by the agent when making decisions, including greedy, linear, constant, normal distribution sampling, OU sampling and other strategies. The core of the training platform is the code implementation of the training paradigm, which is responsible for process resource allocation, status checking, hook parsing, agent cloning, and agent training and evaluation.

[0035] Among them, the configuration plug-in provides functions such as parameter integration, parameter configuration, function module registration, and custom configuration parameters. The main purpose is to facilitate the intelligent agent module and environment module in the training subject, and to facilitate tasks such as fine-tuning parameters, combing parameters, replacing modules, and configuring training. The function plug-in provides configuration file parsing module, hook function, log function, module registration function, training indicator module, intelligent agent evaluation function, flask api, etc. The function plug-in is mainly to combine the experience replay pool and various auxiliary operations to obtain the training paradigm, so as to more conveniently serve the training platform in the training subject.

[0036] Since most reinforcement learning problems can be transformed into Markov decision processes (MDP) problems, we can choose to use Markov decision processes (MDP) to model reinforcement learning. The modeling process is designed to use the Bellman equation to recursively update and evaluate the value of each state, and then gradually approach the optimal strategy. On this basis, the characteristics of value iteration and policy iteration are used to further optimize the decision process. TD (temporal difference algorithm) is an important value function evaluation method in reinforcement learning. It combines the advantages of Monte Carlo and dynamic programming. At the same time, TD can be used for on-policy (online strategy) and off-policy (offline strategy) learning at the same time, meeting the requirements for unified abstraction of current reinforcement learning algorithms. This is also the basic premise for designing a unified reinforcement learning algorithm training paradigm.

[0037] In one embodiment, Figure 2 As shown, a data processing method for reinforcement learning algorithm is provided, and the method is applied to a data processing system for reinforcement learning algorithm, hereinafter referred to as the system, comprising the following steps:

[0038] Step 202, determine the interface function associated with the enhancement algorithm to achieve the adaptation of the target enhancement algorithm and the data environment.

[0039] Specifically, refer to Figure 1 As shown in the system framework structure, for those who need to access a reinforcement algorithm or data environment from scratch, the system provides access interface functions, and only needs to inherit the interface semantics and then implement the corresponding algorithm. Each interface has a clear and strong definition, that is, a unified interface function is provided for the application of intelligent agent algorithms. After the reinforcement algorithm and data environment are accessed and adapted, they can be configured through the configuration file to complete the training task. The whole process is very simple and efficient.

[0040] In one embodiment, the reinforcement algorithm includes a single-agent algorithm and a multi-agent algorithm; the reinforcement algorithm corresponding to the interface function abstracts a multi-layer interface in the system framework structure, including a model layer model, an algorithm layer embryo and an agent layer; wherein: the model layer model is defined using a neural network, and implements the model definition part and the reasoning part; the algorithm layer embryo points to the reinforcement learning algorithm to update the learning part, and the corresponding interface function includes at least an update function update and a prediction function execute; the agent layer represents an entity that interacts with the data environment through code logic, and the corresponding interface function includes at least a learning function learn and a policy function policy.

[0041] Among them, single-agent algorithms include Q-learning, DQN and other algorithms, and multi-agent algorithms include MADDPG, QMix and other algorithms; providing a unified access interface is that the system abstracts the model layer, algorithm layer, and agent layer interface for all algorithms. Figure 1 shown.

[0042] The model layer in the system framework structure is generally defined by the deep neural network, table, etc. adopted by the algorithm. The deep neural network is generally composed of MLP or COV. The interface implementation provided is mainly the model definition part (__init__ function) and the reasoning part (forward function). The framework used is generally pytorch and tensorflow. It also includes a series of interface functions to obtain and set parameter weights, init_params initialization function, parameter synchronization function, etc.

[0043] The algorithm layer "embryo" in the system framework generally refers to the learning part of the reinforcement learning algorithm update. The strategy update and control of the reinforcement learning algorithm are ultimately determined by this layer. The interface functions provided include update, execute, save_memory, get_memory, sync_update_weight, etc. The core interface functions are the update function update and the prediction function execute. Among them, the update interface is used to implement operations such as extracting samples, algorithm updates, and loss back propagation; the execute interface is a prediction function used to predict the probability of the agent taking actions under different environmental conditions; save_memory stores samples (s, a, s', r) in the experience replay pool; get_memory randomly extracts a batch from the experience replay pool (such as taking 32 samples as a batch); sync_update_weight is a strategy update.

[0044] The agent layer in the system framework structure represents the entity that interacts with the environment through code logic, that is, it provides a unified interface for interacting with the data environment, including learn, predict, policy, is_learn, save, resume, update_model_params, and take_model_params functions. The learning function learn is used to directly obtain samples from the environment, determine whether to update the strategy, and whether to learn; is_learn indicates whether the agent is in a state where it can be updated; the predict interface function is an encapsulation of execute, indicating the behavior predicted by the agent in the current state; the policy function policy indicates whether the current agent adopts a deterministic strategy or a random strategy, etc.; save saves the model, resume restores the model, update_model_params updates the model parameters, and take_model_params obtains the model parameters.

[0045] In one embodiment, the system provides a series of RESTful interfaces for training, reasoning, data analysis, and task control, such as train, val, scalar, stop, etc. to connect to external data. Figure 1 The functions of the configuration plug-in shown can be freely connected to other systems as a backend training component for reinforcement learning. Since the provided restful functional interface can also make reinforcement learning training a quick plug-in, whether it is an industry insider or an amateur, this will provide strong support for the quick training of reinforcement learning algorithms.

[0046] Step 204, configure the target parameters and the target reinforcement algorithm to obtain a training task related to the configuration file.

[0047] Specifically, Figure 3 As shown, Figure 3 The figure is a flowchart of accessing an algorithm training task in an embodiment. After the intelligent agent, model and data environment are customized in the system, the target parameters of the algorithm, model and data environment can be configured according to the application. Then, the target parameters are configured and decoupled from the algorithm and the related application code logic to obtain a configuration file, so as to uniformly generate the training tasks of the intelligent agent and data environment through the configuration file. This allows subsequent training to be carried out through the unified training paradigm of the training task, and after the training is completed, the final status of the training can be viewed through the training plug-in, so that the target parameters can be adjusted quickly and conveniently to achieve the effect of optimizing the training.

[0048] Step 206, using temporal difference as the basic process of interaction between the intelligent agent and the data environment, and combining the experience replay pool and multiple auxiliary operations to obtain a training paradigm.

[0049] Among them, the temporal difference (TD) representation does not need to know the complete environment model and can be learned directly from experience. At the same time, it can update the current estimate based on the estimated value function that has been learned. The training of all reinforcement learning algorithms in this system follows a unified training paradigm, which is a standard process based on TD(0) policy evaluation.

[0050] Specifically, this system is based on the methodology of temporal difference and the behavioral characteristics of reinforcement learning itself, which describes the general framework of learning by which the agent interacts with the environment to obtain rewards. In the middle process, the agent hook, parameter parsing, module registration, scalar operation, log and other auxiliary operations and experience playback pool are inserted to unify the training of all reinforcement learning. Such a framework forms a training paradigm, that is, Figure 1 The function of the functional plug-in shown in the figure. This system provides a variety of experience replay pools of different lengths, such as single-step experience replay pool, multi-step experience replay pool, whole-game experience replay pool, adaptive replay pool and priority experience replay pool (PER). These experience replay pools are integrated into the system and can be directly declared and called in the algorithm update interface. They are effective when storing sample interfaces. The external process is basically the same as the training paradigm. The difference lies in the training conditions of the agent and the different types of experience pools, which lead to different timings for sample storage and strategy learning.

[0051] In one embodiment, Figure 4The pseudo code in the figure clearly shows the update process of the entire training paradigm. Initialize the value function b, then poll all episodes, initialize the state at the beginning of each episode d, poll the step in the episode; e, the strategy evaluates the current state and selects an action; f, executes the action, observes the reward and the next state; g, updates the value function according to Bellman; f, updates the current state; h, determines whether the current episode is a terminal state, if so, enters the next episode, if not, continues to poll the next step in this episode.

[0052] Step 208, complete the data processing of the training task through the training paradigm.

[0053] Specifically, during the execution of the training task, the currently trained agent model will be evaluated to further understand the current agent training status. The evaluation method uses a thread pool to complete multiple evaluations in a short time, improving the overall training efficiency. After the training is completed, the training process is displayed, and information such as rewards, steps, losses, and sample size are captured during the training process.

[0054] In the above data processing method for reinforcement learning algorithm, in order to make the switching and use of training tasks simple and convenient, the target parameters and the target reinforcement algorithm are configured, so that the target parameters can be adjusted quickly and conveniently to achieve the effect of optimizing training. By using time series difference as the basic process of interaction between the intelligent agent and the data environment, the modeling of the unified training paradigm is solved. With the help of interface functions, experience replay pool and various auxiliary operations, the reinforcement learning algorithm and the reinforcement learning data environment can be quickly and conveniently accessed, and the algorithm can be efficiently modified, debugged and optimized, reducing the logical burden brought by the algorithm implementation.

[0055] In one embodiment, the system has integrated some classic reinforcement learning algorithms, such as DQN, MADDPG, Qmix and other algorithms, and also integrated some commonly used data environments and application scenarios, such as GYMENV, SMAC, AIRFIGHT and other environments. The system also integrates some neural network models, such as mlp, cnn, AC architecture model, qmix model, etc. It takes almost no time to successfully start a reinforcement learning training task using these existing algorithm modules, and then various modifications and adjustments can be made to make the system highly compatible.

[0056] The specific steps are as follows: Figure 5As shown in the figure, it includes: (1) determining the algorithm (DQN, MADDPG, QMIX, etc.) and environment (GYMENV, SMAC, AIRFIGHT, etc.) to be used; (2) adapting the parameters of the determined algorithm in the configuration file; (3) generating a training task by calling the configuration file and starting training; (4) checking the training status through the log and checking the training effect through tensorboard.

[0057] Therefore, no matter whether the existing algorithm modules, data environment and models are used for training or accessed through algorithm training tasks, they will eventually be presented in the configuration file. This means that a configuration file is both a training task and the parameter modification and adjustment on the configuration file is both the algorithm parameter tuning and algorithm switching, which greatly improves the flexibility of the training tasks.

[0058] In one embodiment, the adaptation of the target reinforcement algorithm and the data environment is achieved, including: determining the environment interface corresponding to the data environment; ensuring the consistency of the semantics of the environment interface when the intelligent agent corresponding to the target reinforcement algorithm interacts with the data environment.

[0059] Among them, the functions of the environment interface at least include environment initialization init, action interaction step, scene rendering render, and environment reset reset.

[0060] Specifically, since the consistency of the interface encapsulation to the agent interaction environment lies in the uniformity of the interface semantics of the data environment that the agent interacts with, for example, for data environments such as atari, smac, and mpe, this system provides interface functions such as init, step, render, reset, close, and seed for interface encapsulation, where init is environment initialization, step is action interaction, render indicates whether to render, reset is to reset the environment, close closes the environment, and seed sets the random seed of the environment. These environment interface functions are generally implemented in data environments, but the interface semantics are inconsistent. Therefore, the system provides a set of environment semantic interfaces for direct packaging, which can be directly and quickly adapted and connected to the system.

[0061] In this embodiment, the adaptation process ensures that the reinforcement learning algorithm can efficiently interact with the data environment, reducing unnecessary calculations and resource consumption, thereby improving the algorithm's operating efficiency. By ensuring the consistency of the environment interface semantics, the accuracy of the agent when performing actions can be ensured.

[0062] In one embodiment, the target parameters and the target enhancement algorithm are configured to obtain a training task related to the configuration file, including: implementing the configuration processing of the target parameters and the target enhancement algorithm, and decoupling the code logic to obtain the configuration file; and processing the configuration file through a registration module to obtain the training task.

[0063] Among them, the target parameters include at least initialization parameters, model parameters, and environment parameters; the registration module includes at least optimizer, learning rate scheduler, and strategy scheduler. The configuration file uniformly manages the agent type, data environment, and model parameters, etc. The function plug-in can also be configured through this configuration file, and the plug-in can be further prioritized for query scheduling if necessary. The registration method can be a decorator operation on the dictionary storage class name.

[0064] Specifically, when the target reinforcement algorithm and data environment are quickly adapted to the interface implementation of this system, the target parameters and other parameters that need to be debugged and optimized can be extracted and uniformly managed in a configurable manner. Configuration is a basic feature of this system to achieve lightweight. The configuration file can be in py, json, or yaml format. The system provides a unified registration module to generate operations for agents and data environment entities by parsing configuration files. For example, AGRNTS, EMBRYOS, and MODELS are module registrars for agents and models, ENV is a module registrar for the environment, and BUFFER is a module registrar for the experience replay pool. In addition, more registration modules such as optimizers, learning rate schedulers, and strategy schedulers can be expanded, which can be set according to user needs.

[0065] In this embodiment, the configuration processing makes it easier for the algorithm to adapt to different data environments and training tasks, because different data environments can be adapted by changing the configuration file without making a lot of modifications to the algorithm itself.

[0066] In one embodiment, a training paradigm is obtained by combining an experience replay pool and a variety of auxiliary operations, including: when the data environment enters a game state, a hook is called to obtain a policy factor and a learning factor; the agent uses the policy factor to call a policy function policy, and determines the trajectory data after interacting with the data environment; the hook before learning the target reinforcement algorithm is called to perform pre-learning preparations; the agent calls the learning function learn according to the trajectory data and the learning factor, and updates the strategy of the target reinforcement algorithm according to the experience replay pool; the hook after learning the target reinforcement algorithm is called to perform post-learning preparations and store training information.

[0067] Among them, multiple auxiliary operations include at least adding hooks and training information to the basic process of time series difference, and adding hooks, training scalars, checkpoints, logs and other auxiliary operations to the training paradigm, which can improve the compatibility of different algorithms and application environments and the visualization function of indicators in the statistical training process. Trajectory data includes the next state, reward and game state; training information includes the number of learning steps, reward value, and number of experience samples.

[0068] Specifically, Figure 6 As shown, Figure 6 A flowchart of a training paradigm is shown in Figure 2.

[0069] Step 1: Call the hook call_hook ("before_train_episode") to first check whether the algorithm has any special operations before the start of a full training, then set the state of the current episode, initialize episode_step, episode_reward = 0, and initialize the environment env.reset().

[0070] Among them, hooks are a mechanism in reinforcement learning that allows custom code or operations to be inserted at specific stages of the algorithm, such as monitoring the learning process, recording data, adjusting parameters, or performing other custom tasks. The current scene represents a series of interactions between the agent and the environment, starting from an initial state until a terminal state is reached.

[0071] Step 2: Determine the game status of the current scene (is_over). If it is False, it means it has not ended. If it is True, it means the game has ended in the current scene.

[0072] Step 3. When is_over = False, enter the game loop, first determine whether environment rendering (env.render) is needed, then perform call_hook("before_train_step") to find out whether the agent has factors that need to be prepared before interacting with the data environment, and then use reduce_factor to extract the policy factor (policy_factor) and learning factor (learn_factor).

[0073] Among them, the strategy factor refers to telling the agent some information when interacting with the environment, such as telling the agent the current state of the environment (status), action latitude (action_dim), etc., and the learning factor refers to the specific elements that the agent needs to include in addition to the elements in the conventional reinforcement learning experience samples (s, a, s', r, done). For example, the DQN experience sample only requires conventional experience samples, while the experience samples of the QMix algorithm also require the effective actions of each agent in the current environment state (agents_avail_actions), the local state of the environment (local_observations), etc.

[0074] Step 4: Call the agent policy function policy, input the policy factor, and output the actions (actions) that the agent finally interacts with the environment after making the policy decision.

[0075] Step 5: The agent interacts with the environment env.step(actions) and returns trajectory data, such as the next state, reward, and game status.

[0076] Step 6: Call the hook call_hook("before_train_learn") to find out if the agent has anything to do before learning. If so, execute it; otherwise, go to step 7.

[0077] Step 7. Call the agent.learn learning function to perform sample storage and learning operations. In the learn learning function, the experience samples will be stored based on the experience replay pool first, and then it will be determined whether the current sample size or sample status meets the learning conditions. For example, learning will be performed only when the sample reaches a certain number or the sample can be learned once every certain number of times it is stored. If the conditions are met, go to step 8, otherwise go to step 9.

[0078] Step 8: Determine whether the conditions for strategy update are currently met, such as how many learning steps are required before a strategy update. If so, the strategy is updated, and then the agent learns (embryo.update), the number of learning steps is increased by 1 (learn_step+=1), and the process goes to step 9; if not, the process goes to step 10.

[0079] Step 9: Call call_hook("after_train_learn") to find out if the agent has anything to do after learning. If so, execute it. If not, go to step 10.

[0080] Step 10: Store training information, such as number of learning steps, reward value, loss, number of experience samples, etc.

[0081] Step 11: Update the variables, including the current game's accumulated reward, the number of training steps plus one, and the number of interaction steps plus one.

[0082] Step 12: Call call_hook("after_train_step") to find out if the agent has anything to do after an interactive learning update. If so, execute it; otherwise, go to step 13.

[0083] Step 13: Use the new status returned during the environment interaction in step 5 to update the game status, and then determine whether the game meets the exit conditions. If so, proceed to step 14; if not, proceed to step 2 and continue the loop.

[0084] Step 14: Print the game information log and call the agent hook call_hook("after_train_episode") to execute what the algorithm needs to do after the game ends.

[0085] During the training paradigm, call_hook is a query execution operation. During the execution process, it will traverse the function plug-ins configured for the current algorithm task. If the sub-function plug-in is currently marked, the function corresponding to the sub-mark will be executed; at the same time, this operation is also the key to the compatibility of different algorithms. For different algorithms, there is a corresponding algorithm hook, which records the operations of each algorithm independent of the training paradigm. For example, the qmix algorithm requires the current effective action latitude and local observation information of each agent in the smac environment during the agent strategy update learning phase. These are special information in the training paradigm and can be supplemented by algorithm operations in the corresponding algorithm hook.

[0086] In this embodiment, by calling the hook to obtain the policy factor and learning factor, these parameters can be dynamically adjusted according to the current data environment state, so that the agent's strategy and learning process are more adapted to the current environment. The agent uses the policy factor to call the policy function policy to determine the trajectory data after interacting with the data environment, which helps the agent learn from the actual interaction and improve the effectiveness of the strategy. And by calling the hook at different stages, the inspection and adjustment mechanism can be inserted into the key steps of the algorithm to improve the adaptability and robustness of the algorithm to different environments and conditions.

[0087] In one embodiment, the strategy of the target reinforcement algorithm is updated according to the experience replay pool, including: determining the type of experience replay pool corresponding to the target reinforcement algorithm and performing initialization operations; determining the current experience bar element of the interaction between the intelligent agent and the data environment, and structuring the current experience bar element through the data structure; when the experience replay pool is a single-step experience pool, learning and updating the intelligent agent until the game state ends.

[0088] Specifically, the training paradigm combines the single-step experience replay pool and the multi-step experience replay pool. The feature is that the experience replay pools used by the algorithms have different functions. For example, DQN uses a single-step experience replay pool, and each interaction with the environment will be collected into experience sample strips; while the QMix algorithm uses a whole game of updated learning samples, and a sample represents the interaction process of a game. Therefore, the intelligent experience will determine the type of experience pool it uses, and the storage experience pool will perform data storage logic judgment, which will not affect the overall training logic. Figure 7 As shown, Figure 7 The figure is a flow chart of storage determination performed by the experience replay pool in the embodiment.

[0089] Step 1: The game starts, the agent is initialized, the type of experience replay pool to be used is determined, the environment is initialized env.reset, the initial status status is obtained, and the game begins.

[0090] Step 2: The agent obtains the state of the environment to make strategic decisions and generate actions.

[0091] Step 3: Use the action env.step to interact with the environment and obtain the current experience bar elements, including the old state, action, reward, new state, whether the game is completed, the number of steps in the previous game, and the learning factor.

[0092] Step 4: Enter the sample storage process and use the data structure Experience to perform structured processing on the current experience bar elements.

[0093] Step 5: When the experience replay pool is a single-step experience pool, add the data structure experience, and if the game is not over, continue to go to step 2 to loop until the game is over.

[0094] In this embodiment, the current experience bar elements are structured and stored in the experience replay pool, which can increase the diversity of samples and avoid excessive sample repetition during training. When the experience replay pool is a single-step experience pool, the agent can perform learning updates until the game state ends, which helps the agent learn from each step of interaction and adjust strategies in time.

[0095] In one embodiment, the above method also includes: when the experience replay pool is a multi-step experience pool and the current step number in the game is not the first step, judging whether the internal serial number of the sample is the game number of this game; if so, adding the elements of this experience bar to the various queues of the multi-step experience pool respectively, and the number of steps in the game is increased by one; if not, performing structured processing on the various queues of the multi-step experience pool, and adding them as experience samples until the game state ends.

[0096] Specifically, refer to Figure 7 As shown, when the experience replay pool in the above step 5 is a multi-step experience pool, it is determined whether the current step number in the game is the first step (cur_episode_index==0). If it is the first step in the game, the multi-step experience pool parameters are initialized.

[0097] Step 6: When the current step number is not the first step, determine whether the internal serial number of the sample (data.index) is the current episode number (cur_episode). If yes, go to step 7; if not, go to step 8.

[0098] Step 7: Add the experience bar element of this time to the internal state queue, action queue, reward queue, new state queue, game state queue and other queues of the multi-step experience pool, increase the number of steps in the game by one (cur_episode_index+1), and go to step 9.

[0099] Step 8: Indicates that the game status is True and completed, performs structured processing on the various queues of the multi-step experience pool and adds them as an experience sample, updates the game number (cur_episode=data.index), initializes the number of steps in the game=0, and goes to step 10.

[0100] Step 9: If the game is not over, continue to loop in step 2.

[0101] Step 10: When the game status ends, the sample storage process also ends.

[0102] In this embodiment, the multi-step experience pool stores information of multiple steps, so that the agent can learn from a longer sequence, which helps to capture the dynamic characteristics of the environment and improve learning efficiency. By judging whether the internal sequence number of the sample is the current game number and adding the experience bar elements of this time to the various queues of the multi-step experience pool, the representativeness of the sample can be ensured, making the learning process more comprehensive and accurate.

[0103] In summary, the time difference method is used to solve the modeling of the unified training paradigm, and some liberal auxiliary operations are performed in the intermediate process to solve the specificity of different algorithms. In combination with the unified agent interface semantics, environment interface, rich plug-ins and configurable parameter file management, it eventually becomes a lightweight system. For off-policy general algorithms learned from data generated by other strategies, etc., it can be flexibly used and the algorithm training tasks can be switched.

[0104] It should be understood that, although the various steps involved in the above embodiments are not necessarily performed in the order indicated. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be performed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0105] Based on the same inventive concept, the embodiment of the present application also provides a data processing system for a reinforcement learning algorithm for implementing the above-mentioned method. The implementation scheme for solving the problem provided by the system is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in the one or more data processing system embodiments for reinforcement learning algorithms provided below can refer to the limitations of the data processing method for reinforcement learning algorithms above, and will not be repeated here.

[0106] In one embodiment, a data processing system for a reinforcement learning algorithm includes: an interface function module, which is used to determine the interface function associated with the reinforcement algorithm to achieve the adaptation of the target reinforcement algorithm and the data environment; a configuration processing module, which is used to configure the target parameters and the target reinforcement algorithm to obtain a training task related to the configuration file; a training paradigm module, which is used to use time series difference as the basic process of interaction between the intelligent agent and the data environment, and combine the experience replay pool and multiple auxiliary operations to obtain a training paradigm; and complete the data processing of the training task through the training paradigm.

[0107] Each module in the above-mentioned data processing system for reinforcement learning algorithm can be implemented in whole or in part by software, hardware and their combination. Each module can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each module above.

[0108] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 8 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store configuration files. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data processing method for a reinforcement learning algorithm is implemented.

[0109] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0110] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0111] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps in the above-mentioned method embodiments.

[0112] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited thereto.

[0113] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0114] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data processing method for reinforcement learning algorithm, characterized in that: Applied in a data environment including an intelligent agent; the method comprises: Determine the interface function associated with the reinforcement algorithm to achieve the adaptation of the target reinforcement algorithm and the data environment; Configuring the target parameters and the target reinforcement algorithm to obtain a training task related to the configuration file; Taking temporal difference as the basic process of interaction between the agent and the data environment, and combining the experience replay pool and multiple auxiliary operations to obtain a training paradigm; The data processing of the training task is completed through the training paradigm.

2. The method according to claim 1, characterized in that: The reinforcement algorithm includes a single-agent algorithm and a multi-agent algorithm; the interface function corresponds to a multi-layer interface abstracted by the reinforcement algorithm, including a model layer model, an algorithm layer embryo and an agent layer; wherein: The model layer model is defined using a neural network and implements the model definition part and the reasoning part; The algorithm layer, Embryo, points to the update learning part of the reinforcement learning algorithm. The corresponding interface functions include at least the update function and the prediction function, Execute. The agent layer represents the entity that performs code logic interaction with the data environment, and the corresponding interface functions include at least a learning function learn and a policy function policy.

3. The method according to claim 1, characterized in that The implementation of the target reinforcement algorithm and the adaptation of the data environment includes: Determine the environment interface corresponding to the data environment; the functions of the environment interface at least include environment initialization init, action interaction step, scene rendering render, and environment reset reset; Ensure the consistency of the environment interface semantics when the intelligent agent corresponding to the target reinforcement algorithm interacts with the data environment.

4. The method according to claim 1, characterized in that: The target parameters at least include initialization parameters, model parameters, and environment parameters; the configuration processing of the target parameters and the target enhancement algorithm to obtain a training task related to the configuration file includes: Implementing configuration processing of decoupling the target parameters from the target enhancement algorithm and the code logic to obtain a configuration file; The configuration file is processed by a registration module to obtain a training task; the registration module includes at least an optimizer, a learning rate scheduler, and a strategy scheduler.

5. The method according to claim 1, characterized in that The various auxiliary operations include at least adding hooks and training information to the basic process of temporal difference; The training paradigm obtained by combining the experience replay pool and multiple auxiliary operations includes: When the data environment enters a game state, a hook is called to obtain a strategy factor and a learning factor; The agent uses the policy factor to call the policy function policy and determines the trajectory data after interacting with the data environment; the trajectory data includes the next state, reward and game state; Call the hook before learning the target reinforcement algorithm to perform preparatory work before learning; The agent calls the learning function learn according to the trajectory data and the learning factor, and updates the strategy of the target reinforcement algorithm according to the experience replay pool; Call the hook after the target reinforcement algorithm is learned, perform the preparatory work after learning and store the training information; the training information includes the number of learning steps, reward value, and number of experience samples.

6. The method according to claim 5, characterized in that The strategy update of the target reinforcement algorithm according to the experience replay pool includes: Determine the type of experience replay pool corresponding to the target reinforcement algorithm and perform initialization operation; Determine the current experience bar element of the agent interacting with the data environment, and perform structured processing on the current experience bar element through the data structure; When the experience replay pool is a single-step experience pool, the agent is learned and updated until the game state ends.

7. The method according to claim 6, characterized in that The method further comprises: When the experience replay pool is a multi-step experience pool and the number of steps in the current game is not the first step, determining whether the internal sequence number of the sample is the game number of the current game; If yes, then add the experience bar elements of this time to the various queues of the multi-step experience pool respectively, and increase the number of steps in the game by one; If not, the various queues of the multi-step experience pool are structured and added as experience samples until the game state ends.

8. A data processing system for reinforcement learning algorithms, characterized in that: include: The interface function module is used to determine the interface function associated with the reinforcement algorithm and realize the adaptation of the target reinforcement algorithm and the data environment; A configuration processing module, used for configuring the target parameters and the target reinforcement algorithm to obtain a training task related to the configuration file; The training paradigm module is used to use the temporal difference as the basic process of interaction between the intelligent agent and the data environment, and to obtain the training paradigm by combining the experience replay pool and a variety of auxiliary operations; the data processing of the training task is completed through the training paradigm.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images