Intelligent agent training method and device, equipment, storage medium and program product

Abstract

The invention discloses an agent training method and device, equipment, a storage medium and a program product, and relates to the technical field of artificial intelligence, and the method comprises the steps: creating a training environment of an agent according to model hyper-parameter configuration; based on a training environment, a shared agent is initialized, the shared agent is provided with a network structure of a core agent and a user-defined optimizer, and the core agent is provided with a multi-head self-attention network and a multi-actor network; in a multi-process parallel environment, carrying out iterative training on the shared intelligent agent according to the input image to obtain an accumulated discount total reward and a total loss size; and performing parameter updating on the shared agent according to the accumulated discount total reward and the total loss. According to the method, the shared agent is optimized by using the network structures of the multi-head self-attention network and the multi-actor network, so that the exploration efficiency of the agent can be improved, local optimum is avoided, and the over-fitting risk is reduced, thereby improving the generalization ability and the sampling accuracy of the agent.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular, to an intelligent agent training method, device, equipment, storage medium, and program product. Background Art

[0002] Reinforcement Learning (RL) is an important branch of machine learning. Through a feedback mechanism, an intelligent agent takes actions to maximize rewards, thereby helping the intelligent agent continuously optimize decisions in complex and uncertain environments to achieve a predetermined goal. Such technologies can be applied to complex decision-making scenarios such as autonomous driving, robot control, and large language models.

[0003] In order to use various types of data to understand real-world tasks such as large models and drones, finding patterns in the analyzed data is the main task. Therefore, it is a key challenge for an intelligent agent to automatically learn high-quality features from a large-scale dataset. For the reinforcement learning of intelligent agents, there are various methods. For example, a non-intrusive load monitoring method based on reinforcement learning and attention mechanism with the publication number CN118364866B, a scalable cluster control method based on multi-agent reinforcement learning with the publication number CN118393880A, a distributed energy system energy scheduling optimization method based on the LSTM-PPO algorithm with the publication number CN118432189A, etc.

[0004] However, when the above reinforcement learning is applied, there are still problems that when the external exploration environment is sparse or the training data is insufficient, it is easy to fall into local optimality and overfitting, resulting in low generalization and accuracy of the intelligent agent. Summary of the Invention

[0005] The main purpose of this application is to provide an intelligent agent training method, device, equipment, storage medium, and program product, aiming to solve the technical problem that when reinforcement learning is applied, it is still easy to fall into local optimality and overfitting when the external exploration environment is sparse or the training data is insufficient, resulting in low generalization and accuracy of the intelligent agent.

[0006] To achieve the above object, this application proposes an intelligent agent training method, and the method includes: Create a training environment for the intelligent agent according to the model hyperparameter configuration; Based on the training environment, initialize a shared intelligent agent. The shared intelligent agent is set with the network structures of a core intelligent agent and a custom optimizer. The core intelligent agent is set with a multi-head self-attention network and a multi-actor network; In a multi-process parallel environment, iteratively train the shared intelligent agent according to the input image to obtain the cumulative discounted total reward and the total loss size of the shared intelligent agent; Update the parameters of the shared agent according to the cumulative discounted total reward and the total loss magnitude.

[0007] In one embodiment, the step of creating a training environment for an agent according to a model hyperparameter configuration includes: Determine the rendering mode status according to the model hyperparameter configuration parsed from the program command line arguments; Based on the agent training environment instance, set the number of random seeds and obtain the number of actions from the action space; According to the number of random seeds, use a random number generator to set a globally fixed seed value; Determine the training environment of the agent according to the rendering mode status, the number of actions, and the seed value.

[0008] In one embodiment, the step of initializing a shared agent based on the training environment includes: Initialize the network structure of the core agent, where the core agent is provided with a feature encoder based on a gated recurrent unit, a multi-head self-attention network, a multi-actor network, and a multi-critic network, and the multi-head self-attention network is used to optimize the multi-actor network; Determine the multi-network structure according to the number of multi-actor networks and the number of multi-critic networks; Based on the current layer number of the multi-network structure, use a linear layer to construct a network and save it to a preset multi-network storage structure, which is used to accommodate multiple networks of the shared agent; Configure the shared memory status of the custom optimizer to complete the initialization of the shared agent.

[0009] In one embodiment, the step of iteratively training the shared agent according to an input image in a multi-process parallel environment to obtain the cumulative discounted total reward and the total loss magnitude of the shared agent includes: In a multi-process parallel environment, loop to create instances of the local training environment for each sub-process, and start the instances of the local training environment for agent training; Copy the core agent as the local model, and determine the local state information according to the local model and the input image; Convert the local state information into a state tensor; Iteratively train the shared agent according to the state tensor to obtain the cumulative discounted total reward and the total loss magnitude of the shared agent.

[0010] In one embodiment, the step of iteratively training the shared agent according to the state tensor to obtain the cumulative discounted total reward and the total loss magnitude of the shared agent includes: Train the shared agent according to the state tensor; When the current round number is less than the preset training round number, share the state information of the local model and the core agent; Determine the value probability weight value output by the multi-critic network according to the information sampling of the interaction between the core agent and the external environment; Use the multi-head self-attention network to weight process the multiple policies output by the multi-actor network to obtain an action policy; Until the current round number reaches the preset training round number, determine the cumulative discounted total reward and the total loss size of the shared agent according to the value probability weight value and the action policy.

[0011] In an embodiment, after the step of updating the parameters of the shared agent according to the cumulative discounted total reward and the total loss size, the method further includes: After the update is completed, end the current subprocess and create a subprocess list; When the current number is less than the current quantity, start each subprocess in a loop to perform asynchronous training on the shared agent until all subprocesses in the subprocess list end; After all subprocesses in the subprocess list end, synchronize the end nodes of all subprocesses.

[0012] In addition, to achieve the above object, the present application also proposes an agent training device, the device includes: A training environment module for creating a training environment for an agent according to model hyperparameter configuration; An agent module for initializing a shared agent based on the training environment, the shared agent being provided with a network structure of a core agent and a custom optimizer, and the core agent being provided with a multi-head self-attention network and a multi-actor network; An iterative training module for iteratively training the shared agent according to an input image in a multi-process parallel environment to obtain the cumulative discounted total reward and the total loss size of the shared agent; A parameter update module for updating the parameters of the shared agent according to the cumulative discounted total reward and the total loss size.

[0013] In addition, to achieve the above object, the present application also proposes an agent training device, the device includes: a memory, a processor, and a computer program stored on the memory and executable on the processor, the computer program being configured to implement the steps of the agent training method as described above.

[0014] In addition, to achieve the above object, the present application also proposes a storage medium, which is a computer-readable storage medium. A computer program is stored on the storage medium, and when the computer program is executed by a processor, the steps of the agent training method described above are implemented.

[0015] In addition, to achieve the above object, the present application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps of the agent training method described above are implemented.

[0016] One or more technical solutions proposed by the present application have at least the following technical effects: The agent training method of the present application includes: creating a training environment for the agent according to model hyperparameter configuration; initializing a shared agent based on the training environment, where the shared agent is provided with a network structure of a core agent and a custom optimizer, and the core agent is provided with a multi-head self-attention network and a multi-actor network; in a multi-process parallel environment, iteratively training the shared agent according to the input image to obtain the cumulative discounted total reward and the total loss size of the shared agent; updating the parameters of the shared agent according to the cumulative discounted total reward and the total loss size. Since the present application uses the network structures of the multi-head self-attention network and the multi-actor network to optimize the shared agent, it can improve the exploration efficiency of the agent, avoid falling into local optima, reduce the risk of overfitting, and thus improve the generalization ability and sampling accuracy of the agent. Description of the Drawings

[0017] The drawings here are incorporated into the specification and form a part of the specification, showing embodiments consistent with the present application and used together with the specification to explain the principles of the present application.

[0018] To more clearly illustrate the technical solutions in the embodiments of the present application or in the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0019] Figure 1 It is a schematic flowchart provided for Embodiment 1 of the agent training method of the present application; Figure 2 It is a schematic diagram of the principle of agent sampling combining a multi-head self-attention network and a multi-network structure provided for Embodiment 1 of the present application; Figure 3 It is a schematic diagram of decision optimization combining a multi-actor network and a multi-head self-attention network provided for Embodiment 1 of the present application; Figure 4 It is a schematic flowchart provided for Embodiment 2 of the agent training method of the present application; Figure 5 This is the overall implementation flowchart of agent training provided in the second embodiment of the present application; Figure 6 This is the detailed process schematic diagram of agent training provided in the second embodiment of the present application; Figure 7 This is the module structure schematic diagram of the agent training device in the embodiment of the present application; Figure 8 This is the device structure schematic diagram of the hardware operating environment involved in the agent training method in the embodiment of the present application.

[0020] The realization, functional characteristics, and advantages of the present application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. Specific embodiments

[0021] It should be understood that the specific embodiments described herein are only used to explain the technical solution of the present application and are not used to limit the present application.

[0022] In order to better understand the technical solution of the present application, the following will be described in detail in conjunction with the accompanying drawings of the specification and specific embodiments.

[0023] It should be noted that for reinforcement learning, in traditional methods, a method of reinforcement learning with human feedback (RLHF) for fine-tuning large models has been further proposed, that is, by combining human feedback, the agent can gradually optimize its dialogue strategy or generation strategy, enabling it to make better decisions in multi-round interactions; for example, the agent can learn how to better respond to user needs or provide more relevant and coherent answers.

[0024] And in order to further improve the performance of the agent in high-dimensional complex environments, traditional methods have also proposed a deep reinforcement learning method (DRL) that combines the decision-making ability of reinforcement learning and the feature extraction ability of deep learning technology, that is, by approximating the Q function through a deep neural network, the agent can make effective decisions from unstructured data.

[0025] Among them, the sampling strategy for the agent can be divided into three types: (1) Value-oriented agent: It learns the strategy by updating the state-action value function, usually relying on the greedy algorithm for data sampling, randomly selecting a part of the actions to explore new state information to balance the exploration and exploitation relationship of the agent.

[0026] (2) Policy-oriented agent: It learns to make optimal decisions by optimizing the parameters of the policy (i.e., policy gradient) to output the probability distribution of actions. It usually uses probability sampling, mainly including deterministic and stochastic methods. The agent selects actions according to the current policy probability distribution to promote the exploration of new environments.

[0027] (3) Agent that fuses value and policy: It uses an Actor network to select actions and a Critic network to evaluate actions, and uses the Temporal Difference (TD) method to update the network simultaneously, combining the advantages of value-based and policy-based methods. Such agents generally use an online update method at each timestamp to adjust their policies. The mainstream sampling methods include TD sampling, multi-step TD sampling, and trajectory segment sampling, etc.

[0028] Generally speaking, the use of deep learning models and the introduction of deep reinforcement learning have brought practical improvements. It is generally applied to practical tasks such as video games, action recognition, and face recognition. For example, in video games or simulation environments, a Convolutional Neural Network (CNN) is usually used to extract features from input pixels. In time series data tasks, a Recurrent Neural Network (RNN) or a Long Short-Term Memory Network (LSTM) is used to extract time-dependent features. Optimizing the feature extraction module of the agent using a deep network is beneficial to extract effective state and action information from the environment, thereby helping the agent select optimal decisions and improve sampling efficiency and performance.

[0029] However, in the prior art, there are the following problems for the reinforcement learning of agents: (1) For example, for a non-intrusive load monitoring method based on reinforcement learning and attention mechanism, this method optimizes the load monitoring accuracy of a small parameter model by combining the Deep Deterministic Policy Gradient (DDPG) algorithm and the attention mechanism to replace the traditional deep model. However, in the training process of this method, the target network and the evaluation network are prone to inconsistent updates and gradient explosion, resulting in problems such as slow model convergence speed and poor training stability.

[0030] (2) For example, for the scalable cluster control method based on multi-agent reinforcement learning, this method uses the self-attention mechanism to optimize the evaluation network model in the multi-agent algorithm, aiming to extract the key state information between agents. This method relies on the agent's ability to obtain sufficient data information from the outside. When the external exploration environment is sparse or the training data is insufficient, it is easy to fall into local optimum and cause problems such as overfitting and low generalization ability.

[0031] (3) For example, for the distributed energy system energy scheduling optimization method based on the LSTM-PPO algorithm, the LSTM network is used to embed the Proximal Policy Optimization (PPO) algorithm, and the feature information extracted by the network is used as the input of the Proximal Policy Optimization agent (PPO) to improve the ability to handle uncertain factors. However, the structure of the LSTM model is complex and has many parameters, which easily leads to problems such as a large number of agent parameters, slow training speed, and high computational resource consumption.

[0032] Therefore, to solve the above problems, this application proposes an agent training method, and the specific implementation process is as follows.

[0033] It should be noted that the execution subject of this embodiment can be an electronic device with functions such as training environment construction, agent setting, and iterative training, such as a personal computer, a server, etc., or an electronic device capable of implementing the above functions, an agent training device for executing the agent training method of this application, etc. This embodiment does not limit this. Hereinafter, taking the agent training device as an example, this embodiment and the following embodiments will be described.

[0034] Based on this, Embodiment 1 of this application is proposed. Embodiment 1 of this application provides an agent training method, referring to Figure 1 , Figure 1 is the flowchart provided for Embodiment 1 of the agent training method of this application.

[0035] In this embodiment, the agent training method includes steps S10~S40: Step S10: Create a training environment for the agent according to the model hyperparameter configuration.

[0036] It should be noted that an agent can be an entity in deep learning that can make decisions and take actions based on the perceived information. An agent can be a software program. For example, in reinforcement learning, an agent takes actions in the environment, receives feedback from the environment, and then adjusts its decision-making strategy according to the feedback.

[0037] It is understandable that the model hyperparameter configuration can be the parameter settings required for the operation of the agent during its operation. Exemplarily, the model hyperparameter configuration may include, but is not limited to, the learning rate (which determines the speed of model learning), the batch size (the amount of data processed each time), the number of iterations (the number of rounds of model training), etc. This embodiment does not limit this.

[0038] It should be understood that the training environment can be the operating scenario in which the agent is located during the process of learning and optimizing the decision-making strategy. In the training environment, the agent can gradually optimize its behavior strategy through continuous trial and error and learning.

[0039] In a specific implementation, the agent training device can first obtain the model hyperparameter configuration required for the operation of the agent during its operation. Then, according to this model hyperparameter configuration, a training environment in which the agent is located during the process of learning and optimizing the decision-making strategy is constructed.

[0040] In a feasible implementation manner, step S10 of this embodiment may include the steps of: determining the rendering mode state according to the model hyperparameter configuration parsed from the program command line parameters; setting the number of random seeds and obtaining the number of actions from the action space based on the agent training environment instance; setting a globally fixed seed value using a random number generator according to the number of random seeds; and determining the training environment of the agent according to the rendering mode state, the number of actions, and the seed value.

[0041] It should be noted that the program command line parameters can be multi-threaded environment variables manually configured or obtained by the agent before running, and these variables are input into the agent training device in the form of a program command line. The agent training device parses these program command line parameters and can convert them into model hyperparameter configurations that can be understood and used within the program.

[0042] It is understandable that the rendering mode state can be the situation of whether the rendering mode is started in the training environment. Exemplarily, the environment rendering parameters in the model hyperparameter configuration can be extracted, and whether to start the rendering mode is judged according to the environment rendering parameters; if the environment rendering parameters are true, the rendering mode is started, the number of processes is set to 1, the agent test mode is entered, and the model learning rate is set to 0 to ensure that the weights and parameters are not updated during the model test process and remain unchanged; if it is false, the training mode and parameters remain unchanged.

[0043] Exemplarily, the number of threads in the parallel training area of the agent can be 1, and the model hyperparameter configuration to ensure the operation of the agent can include the agent training environment name, the number of threads can be 20, the number of random seeds can be 1, the learning rate can be 10-4, the environment rendering parameters, and the test mode parameters, etc.; It should be understood that the intelligent agent training environment instance can be a parallel training environment instance created for the intelligent agent to interact and learn in the scenario of reinforcement learning. The action space can be the set of all possible actions that the intelligent agent can take in the training environment.

[0044] The number of actions (num_actions) can be the total number of actions that the intelligent agent may take in the given training environment action space. Among them, the number of actions is mainly used for the construction of the output layer of the subsequent multi-actor home network.

[0045] It should be noted that the number of random seeds can be the number of random seeds set in the training environment when there are scenarios involving random processes (such as random data generation, random initialization of model parameters, random selection of initial states, etc.) in order to reproduce the results or control the influence of random factors. The random number generator can be a tool for generating a sequence of random numbers.

[0046] The globally fixed seed value is a fixed and unchanging seed value used throughout the training environment to ensure that the random data sequence for each training of the intelligent agent is the same.

[0047] In this embodiment, first, the program command line parameters can be parsed to obtain the model hyperparameter configuration, and the intelligent agent directory can be set according to the model hyperparameter configuration; then, it is judged whether to start the rendering mode according to the environment rendering parameters extracted from the model hyperparameter configuration. If the rendering parameter is true, the rendering mode is started; if it is false, the training mode and parameters remain unchanged. Then, an intelligent agent parallel training environment instance is created, the number of random seeds is set, and the number of actions of the intelligent agent is obtained from the action space of the training environment. Then, it is judged whether the intelligent agent directory exists. If the intelligent agent directory does not exist, an intelligent agent save directory is created; if it exists, a globally fixed seed value is set using the random number generator according to the number of random seeds to ensure that the random data sequence for each training of the intelligent agent is the same. Thus, based on the above settings, the creation of the intelligent agent training environment is completed. Through the creation of the above training environment, it can be ensured that under the same program running conditions (using the same globally fixed seed value), the same random number results can be obtained.

[0048] Step S20: Initialize a shared intelligent agent based on the training environment. The shared intelligent agent is provided with the network structures of a core intelligent agent and a custom optimizer. The core intelligent agent is provided with a multi-head self-attention network and a multi-actor home network.

[0049] It should be noted that the shared intelligent agent (Share_Agent) can be an intelligent agent that can be shared and used by multiple different tasks, modules, or subsystems. Compared with creating an intelligent agent separately for each task, the shared intelligent agent can improve resource utilization and reduce the repetitive labor of model training.

[0050] Among them, the Core_Agent can be the main component of the shared agent, which includes the implementation of the most basic functions of the agent, such as decision-making and learning. The Core_Agent may include structures such as deep neural networks. For example, the Q-network part in the Deep Q-Network (DQN) can be regarded as part of the Core_Agent. It outputs the value estimate of each possible action based on the input environmental state, thereby helping the agent make decisions.

[0051] It can be understood that the multi-head self-attention network can be a neural network structure based on the attention mechanism, which is used to enable the Core_Agent to focus on different parts of the input. The multi-actor network can be a network structure with multiple actor modules in the system of the Core_Agent. Among them, each actor can be regarded as an independent decision-making unit, responsible for making different decisions in the environment.

[0052] It can be understood that the Agent_Optimizer can be an optimization algorithm structure specifically designed for the Core_Agent. The Agent_Optimizer can adjust the parameters of the Core_Agent according to the loss function to improve the performance of the model. The input parameters of the Agent_Optimizer can be set as all the parameters of the Core_Agent, including the weights and biases of model updates, as well as the learning rate of the model.

[0053] In a specific implementation, after determining the training environment, the Core_Agent can be constructed according to the multi-head self-attention network and the multi-actor network, and then a network structure can be constructed together with the Agent_Optimizer to obtain the shared agent, and the shared agent can be initialized.

[0054] In a feasible implementation manner, step S20 of this embodiment may include the steps of: initializing the network structure of the Core_Agent, where the Core_Agent is provided with a feature encoder based on a gated recurrent unit, a multi-head self-attention network, a multi-actor network, and a multi-critic network, and the multi-head self-attention network is used to optimize the multi-actor network; determining the multi-network structure according to the number of the multi-actor network and the number of the multi-critic network; based on the current layer number of the multi-network structure, constructing a network using a linear layer and saving it to a preset multi-network storage structure, where the multi-network storage structure is used to accommodate multiple networks of the shared agent; configuring the shared memory state of the Agent_Optimizer to complete the initialization of the shared agent.

[0055] Exemplarily, taking the Core_Agent with four parameters as an example for illustration, the network parameters of the Core_Agent can be configured such that both the hidden layer size and the input data size are 256, the number of input channels is 1, the number of layers of the multi-actor network can be a variable n, and the number of layers of the multi-critic network can be a variable m. For example, n can take 4 and m can take 2.

[0056] It should be noted that a feature encoder based on a Gated Recurrent Unit (GRU) can be used to extract key features of input pixels. By using a GRU network, the key information between features can be captured to improve the training speed of the agent.

[0057] Exemplarily, for ease of understanding, refer to Figure 2 , Figure 2 FIG. is a schematic diagram of agent sampling combining a multi-head self-attention network and a multi-network structure provided in Embodiment 1 of the present application. The feature encoder based on the gated recurrent unit can be sequentially composed of four cascaded two-dimensional convolutional layers (such as Conv2d1, Conv2d2, Conv2d3, and Conv2d4) and a gated recurrent unit (GRU). Among them, the network parameters of the first two-dimensional convolutional layer Conv2d1 can be set as follows: the number of input channels is 1, the number of output channels is 32, the convolutional kernel size is 3×3, the convolutional stride is 2, and the edge padding size is 1; the network parameters of the second, third, and fourth two-dimensional convolutional layers Conv2d2, Conv2d3, and Conv2d4 can be the same, the number of input channels can be 32, and the size of the finally output feature map is 5×5; the number of input features of the GRU network can be set to 800, and the hidden layer size of the GRU unit is 256.

[0058] It can be understood that the multi-critic network can be used to evaluate the value between diverse policies. By enhancing the evaluation ability of the model, the variance of value estimation can be reduced, thereby improving the accuracy of policy evaluation.

[0059] The multi-actor network is used to output diverse policies. By enhancing parallel training, the policy optimization process can be accelerated, thereby improving the exploration ability of the model in complex environments.

[0060] Among them, the multi-head self-attention network can be used to optimize the multi-actor network. By calculating the dependencies between the diverse policies output by the parallel multi-actor network, the accuracy and generalization of the policies can be improved. Exemplarily, this network initializes three linear transformation weights , and , respectively performs spatial mapping to generate queries, keys, and values, and uses the multi-head attention output layer to recombine the outputs of all heads into a single output tensor, setting its input dimension and output dimension to be the same size.

[0061] It should be understood that the multi-network structure can be the network structure set by the number of multi-actor networks and the number of multi-critic networks, and is used to realize data augmentation in the agent sampling process. Exemplarily, such as Figure 2As shown, the multi-critic network and multi-actor network in the multi-network structure can be set to a four-layer structure, including an input layer, a hidden layer, a linear layer, and an output layer. The output parameters can include the hidden layer size, the input layer channel value, and the output layer channel value. The output channel of the multi-actor network is the above-mentioned number of actions, and the output channel of the critic network is 1.

[0062] It should be noted that the multi-network storage structure can be a storage architecture that can accommodate multiple networks of shared agents. The Linear layer can be a basic neural network layer in the field of neural networks, used for simple linear mapping of input data.

[0063] Specifically, the specific steps of constructing a network using the linear layer based on the current layer number of the multi-network structure and saving it to a preset multi-network storage structure can include: First, create a multi-network storage structure and execute the first-layer loop: Use the linear layer to construct the first-layer network Net1 and save it to the multi-network storage structure (where the input dimension of the network Net1 can be set to 256, and different parameter selections are made for the output dimension size according to the constructed actor or critic network type). Then execute the second-layer loop: Obtain the number of networks , when starting to construct the second network Net2, execute the first-layer loop process until the first preset condition is met.

[0064] Exemplarily, the first-layer loop process can include: Obtain the first-layer loop control parameter ; Compare the first-layer loop control parameter and the network layer number N; where, as Figure 2 shown, the range of the first-layer loop control parameter is [1, N], and the first preset condition is to compare that the first-layer loop control parameter is less than the network layer number N, initialize the network storage list, and first construct the input linear layer Linear1 of the + 1-th network Net + 1 and save it to the storage list; then construct the hidden layer of the + 1-th network Net + 1, and execute the second-layer loop process until the second preset condition is met.

[0065] The second-layer loop process can include: Obtain the second-layer loop control parameter ; Obtain the second-layer loop control parameter and the first-layer loop control parameter ; where the range of the second-layer loop control parameter is [0, - 1]. The second preset condition is to compare the second-layer loop control parameter Less than the first-layer loop control parameter When subtracting 1, construct a linear layer Linear +1 as the network Net +1 hidden layer and save it to the storage list, update the control parameter Add 1, the hidden layer size can be set to 256; then use the linear layer to construct the network Net +1 output layer, its input dimension can be set to 256 and the output dimension to the number of actions or 1; the +1st network Net +1 construction is completed, saved to the multi-network storage structure, and update the control parameter Add 1.

[0066] In this embodiment, first initialize and construct the network structure of the core agent, including a feature encoder based on a gated recurrent unit, a multi-head self-attention network, a multi-actor network, and a multi-critic network. Then construct a multi-network structure according to the number of actor networks and critic networks, including a multi-actor network structure and a multi-critic network structure. Next, create a multi-network storage structure, use a linear layer to construct the first-layer network and save it to the storage structure, then obtain the number of networks. When starting to construct the second network, execute the first-layer loop process until the first preset condition is met. Finally, initialize the custom optimizer of the core agent, traverse the parameter groups of the custom optimizer, initialize a specific shared memory state state for the parameters in each parameter group, and configure the shared memory among multiple processes in the custom optimizer so that multiple processes can access and update the same state variables during the parallel training process. Thus, the initialization of the shared agent is completed. Through the above initialization of the shared agent, the gated recurrent unit can be used to optimize the state encoder of the agent, reduce the number of parameters during the agent training process, improve the agent training speed, and reduce the consumption of computing resources. At the same time, using a parallel network structure to optimize the agent speeds up the efficiency of the agent in collecting sample information and speeds up the speed of the agent in finding the global optimal strategy, which can prevent the agent from overfitting and improve the generalization ability of the agent.

[0067] Further, when traversing the parameters in each parameter group, actual gradient update calculations can be performed, and shared information (which can include average reward, loss size, number of agent training episodes Episodes, and number of training frames Frames) can be initialized to the shared memory for sharing among multiple processes.

[0068] Step S30: In a multi-process parallel environment, iteratively train the shared agent according to the input image to obtain the cumulative discounted total reward and total loss size of the shared agent.

[0069] It should be noted that the cumulative discounted total reward can be the sum of all rewards obtained by the agent through completing specific actions during the entire interaction process of the shared agent. This total reward comprehensively considers the benefits obtained from its interactions with other agents and the environment at various stages, and is the cumulative result after discount processing.

[0070] The magnitude of the total loss can be used to measure the difference between the prediction result and the true result of the agent during the training process of the shared agent, reflecting the overall poor fitting degree of the shared agent on the training dataset.

[0071] Specifically, in a multi-process parallel environment, the parallel training process of the shared agent can be executed. The shared agent is iteratively trained according to the input image. Different tasks are assigned to different sub-processes for local model configuration synchronization status information, and information sampling is performed using the multi-head self-attention network and multi-network structure. Finally, the cumulative discounted total reward and the magnitude of the total loss of the shared agent are obtained.

[0072] In a feasible implementation manner, step S30 of this embodiment may include the steps of: in a multi-process parallel environment, circularly create instances of the local training environment for each sub-process, and start the instances of the local training environment for agent training; copy the core agent as the local model, and determine the local state information according to the local model and the input image; convert the local state information into a state tensor; perform iterative training on the shared agent according to the state tensor to obtain the cumulative discounted total reward and the magnitude of the total loss of the shared agent.

[0073] It should be noted that a sub-process can be one of multiple processes created by the parent process in a multi-process parallel environment for sharing different tasks for parallel processing.

[0074] Exemplarily, the number of sub-processes P can be 20, and the unique identifier of the sub-process P_flag can be , and the functions involved in the training process include the core agent and a custom optimizer.

[0075] Among them, during the process of creating instances of the local training environment for each sub-process and starting the instances of the local training environment for agent training, the random seed (local_seed) of the local environment can be obtained as the sum of the global seed and the process identifier P_flag. Using the random number generator to set the random seed of the local environment can ensure the repeatability of network initialization and other random operations.

[0076] It can be understood that the local state information can be information used to describe the current state of the local model. The state tensor can be data obtained by converting the local state information into a tensor form.

[0077] Furthermore, as Figure 2 shown, in the intelligent agent interaction environment, after receiving the above input image, image preprocessing can be performed on the above input image. The image preprocessing operations can include: First, the input image can be cropped vertically, and then the average value can be calculated along the direction where the channel dimension of the image is 2 to convert the RGB color image into a grayscale image. Then, the size of the grayscale image is adjusted to pixels, and then it is converted to the standard type format (such as float32 format) and image reshaping operation. Finally, the pixel values are divided by 255 for normalization operation to obtain the processed standardized image. After obtaining the standardized image after image preprocessing, a gated recurrent unit feature encoder can be used to reduce the number of parameters in the intelligent agent training process, and the obtained data state is input into a multi-network structure constructed by a multi-critic network and a multi-actor network for subsequent processing.

[0078] Among them, the grayscale image conversion formula in the image preprocessing operation is as follows: ; Among them, represents the cropped image, represents the image width, represents the image height, c = 2 is the number of channels, and are the sizes of the cropped image, and can both be 80.

[0079] In this embodiment, first, instances of the local training environment for each subprocess are created cyclically, and the subprocesses are started for intelligent agent training. Then, the core intelligent agent is copied as the local model (local_model), the environment is reset to obtain the initial local state information (local_state), the input image is preprocessed to obtain the local state information, and it is converted into a state tensor as the input of the intelligent agent. Finally, the shared intelligent agent is iteratively trained according to the state tensor. Thus, by converting to a state tensor, it helps to perform subsequent calculations in the intelligent agent of deep learning.

[0080] In another feasible implementation, the step of iteratively training the shared agent according to the state tensor to obtain the cumulative discounted total reward and the total loss magnitude of the shared agent includes: training the shared agent according to the state tensor; when the current round number is less than the preset training round number, sharing the state information of the local model and the core agent; determining the value probability weight value output by the multi-critic network according to the information sampling of the interaction between the core agent and the external environment; using the multi-head self-attention network to weight-process the multiple policies output by the multi-actor network to obtain an action policy; until the current round number reaches the preset training round number, determining the cumulative discounted total reward and the total loss magnitude of the shared agent according to the value probability weight value and the action policy.

[0081] Specifically, for ease of understanding, an example is given below. First, start training the agent until the number of training frames reaches Or enter the agent test mode if the test mode parameter is true. Among them, the state information of the local model and the core agent can be synchronized to ensure that all processes use the latest network parameters, reset the agent and obtain the hidden state; if the current training round ends, reset the hidden state of the GRU unit; if not, disconnect the previous state and continue to execute.

[0082] Then initialize the round data storage list for each round, including the value estimates values, action policies policys, actions actions, and rewards rewards of the agent. An internal loop with a specified number of rounds of 1000 can be executed, and the agent will interact with the external environment once in each round.

[0083] Next, use the core agent to interact with the external environment to sample the agent's information and call the multi-critic network. As Figure 2 shown, sequentially traverse the m critic networks to obtain the set of value estimates output, perform tensor conversion and dimensionality reduction operations on the set, and then perform value probability estimation on the set. The obtained value probability weight value is used as the final value estimate .

[0084] Among them, the value estimation based on the multi-critic network The calculation formula is as follows; ; Among them, represents the number of multi-critic networks, represents the th critic network; Among them, the agent first calls the feature encoder based on the gated recurrent unit to perform state encoding on the reshaped state tensor to obtain the state encoding And it is defined as the optimized hidden state of the agent, and the state encoding is input into the subsequent action decision network for advantage evaluation.

[0085] Exemplarily, the reshaped four-dimensional state tensor can be set to [1, 1, 80, 80]. The four dimensions respectively represent the training batch size, the number of channels, the image height, and the image width. The reshaped state tensor and the hidden state (i.e., the state hidden quantity) are used as the inputs of the multi-actor home network. The hidden state is a tensor from the previous round, storing the hidden state of the neural network, and is used to capture the dependency information between state sequences.

[0086] After obtaining the above value estimation, action decision-making begins, which can refer to Figure 2 and Figure 3 , Figure 3 is the schematic diagram of decision optimization combining the multi-actor home network and the multi-head self-attention network provided in the first embodiment of this application. First, the action decision network composed of the multi-head self-attention network and the multi-actor home network can be called, and the output action set distribution (such as action Q, action K, action V) obtained by sequentially traversing n actor home networks is used as the input of the multi-head self-attention network. It is input into the dot-product attention network via a linear layer (such as LinearQ, LinearK, LinearV), and the weight information between action distributions is captured by focusing on the different weights of different heads (such as matrix multiplication Matmul, scaled dot-product attention Scale Dot-product, mask Mask, normalization Softmax, and again matrix multiplication Matmul) of the dot-product attention network, and the final action distribution (Action_log) is calculated. It is normalized ( ) to obtain the action policy , and an action action is randomly sampled from the action policy.

[0087] Among them, the calculation formula for the agent's action decision optimized by combining the multi-actor home network and the multi-head self-attention network is as follows: ; Among them, represents the set of action distributions output by multiple actor home networks, represents the multi-head self-attention network, represents the final action distribution, represents the action policy combining the multi-actor home network and the multi-head self-attention mechanism, represents the multi-critic network.

[0088] Then, execute this action in the agent interaction environment to obtain a new state and a reward, and perform value evaluation; at the same time, perform image preprocessing operations on the new state to obtain the processed state information , the cumulative reward calculates the total reward Reward for the current round; Finally, it is determined whether to end the training of a round or the number of rounds exceeds 1000. If true, the average reward and loss size of this round are calculated. If false, the value estimates values, optimal policies policys, actions actions, and rewards rewards are stored in the round data storage list and the current agent model is saved regularly.

[0089] In this embodiment, by combining the decision optimization of the multi-actor network and the multi-head self-attention network, the number of data samplings of the agent can be enhanced, the dependence between the output policies of the actor networks in different representation subspaces can be captured, and effective decisions can be extracted, thereby improving the sampling accuracy of the agent and the ability to select different policies, preventing getting stuck in local optima, and improving the convergence speed and training stability of the agent.

[0090] Step S40: Update the parameters of the shared agent according to the cumulative discounted total reward and the total loss size.

[0091] Specifically, after obtaining the average reward and loss size of each round through the above training, the cumulative discounted total reward of the next state and the total loss size of the agent training period can be calculated and saved, and at the same time, the agent parameters are updated.

[0092] Among them, the cumulative discounted total reward of the agent at time step t The calculation formula is as follows; ; Among them, represents the immediate reward at time step , is the training round, is the reward discount factor, and step can be 1000, indicating the number of rounds per round of training.

[0093] It should be noted that the total loss size of a single round of training of the agent can be composed of the sum of the policy loss, value loss, and entropy loss.

[0094] The policy loss of the agent is used to measure the gap between the actions selected by the agent under the current policy and the optimal actions to optimize the agent's decision-making. The calculation formula is as follows: ; Among them, represents the advantage function optimized based on the multi-critic network, represents the state value function, represents the action value function of the th critic network.

[0095] Value loss of the agent Used to evaluate the difference between the state value function estimated by the agent and the actual reward. The calculation formula is as follows: ; Where, represents the local new state information at time step t, represents the cumulative discounted total reward at time step t.

[0096] Entropy loss of the agent By increasing the reward for uncertainty, the agent is encouraged to explore more in the early stage of training to avoid premature convergence to sub-optimal policies. The calculation formula is as follows: ; In summary, the calculation formula for the total loss size of the agent is as follows: ; Where, represents the total number of training rounds of the agent, represents the local new state information, represents the total loss of the agent in a single training round, , respectively represent the different weights of value loss and entropy loss.

[0097] Finally, the gradients in the custom optimizer can be cleared, the total loss size can be backpropagated, and the gradients of each parameter can be calculated. Then, the parameters of the shared agent can be updated so that all processes can perform the next update based on these parameters.

[0098] It should be noted that during the agent training process, the maximum length of the gradient can be limited to 40 to prevent gradient explosion. It is necessary to traverse the parameters of the local model and the core agent, synchronize the gradients to the shared agent, and ensure that all processes train on the same parameters.

[0099] In the specific implementation, according to the above iterative training, the cumulative discounted total reward and the total loss size of the agent can be calculated, and finally the gradients can be synchronized to the shared agent for parameter update.

[0100] In the technical solution provided in this embodiment, the agent training device can first obtain the model hyperparameter configuration required for the operation of the agent during its operation. Then, according to the model hyperparameter configuration, a training environment in which the agent is located during the process of learning and optimizing the decision-making strategy is constructed. After determining the training environment, a core agent can be constructed based on the multi-head self-attention network and the multi-actor network, and then a network structure can be constructed together with a custom optimizer to obtain a shared agent, and the shared agent is initialized. Then, in a multi-process parallel environment, the parallel training process of the shared agent is executed, and the shared agent is iteratively trained according to the input image. Different tasks are assigned to different sub-processes for local model configuration synchronization status information, and information sampling is performed by the multi-head self-attention network and the multi-network structure. According to the above iterative training, the cumulative discounted total reward and the total loss size of the agent can be calculated, and finally the gradients are synchronized to the shared agent for parameter update. Since this embodiment uses the network structures of the multi-head self-attention network and the multi-actor network to optimize the shared agent, the exploration efficiency of the agent can be improved, the risk of falling into local optimality can be avoided, the overfitting risk can be reduced, and thus the generalization ability and sampling accuracy of the agent can be improved.

[0101] Based on the above-mentioned Embodiment 1 of the present application, Embodiment 2 of the present application is proposed. In the second embodiment of the present application, the same or similar content as that in the above-mentioned Embodiment 1 can be referred to the above introduction and will not be repeated hereinafter. On this basis, please refer to Figure 4 , Figure 4 which is a schematic flowchart provided for Embodiment 2 of the agent training method of the present application.

[0102] After step S40 in this example, steps S41 to S43 are further included: Step S41: After the update is completed, end the current sub-process and create a sub-process list.

[0103] It should be noted that the sub-process list is a set used to store a series of sub-process related information in a multi-process environment. When a parent process creates multiple sub-processes, in order to facilitate the management of these sub-processes, a sub-process list can be created.

[0104] Exemplarily, each element of the sub-process list can include the identifier of the sub-process, the running status of the sub-process (such as running, paused, ended, etc.), the resource information occupied by the sub-process, and other related contents.

[0105] Step S42: When the current number is less than the current quantity, loop to start each sub-process and perform asynchronous training on the shared agent until all sub-processes in the sub-process list end.

[0106] Step S43: After all sub-processes in the sub-process list end, synchronize the end nodes of all sub-processes.

[0107] It should be noted that the current number can be the number of a subprocess in the subprocess list, and the current quantity can be the total number of all subprocesses in the subprocess list.

[0108] In this embodiment, after updating the parameters of the shared agent according to the cumulative discounted total reward and the magnitude of the total loss, the process is first terminated and a subprocess list is created to facilitate subsequent management of the subprocesses. Then, multiple independent subprocesses are cyclically started for asynchronous training of the agent. Each process shares the same model, optimizer, and parameters and is added to the subprocess list. Finally, the agent call program is blocked until all corresponding subprocesses have completed execution, thereby ensuring that the main process will wait for all subprocesses to finish executing before continuing to execute new tasks and synchronizing the end nodes of all processes. Thus, through the subprocess list, the situation of these subprocesses can be better grasped and operated, improving the efficiency of agent training.

[0109] Exemplarily, to facilitate understanding of the implementation process of the agent training method obtained by combining the above Embodiment 1 and Embodiment 2 of this embodiment, please refer to Figure 5 and Figure 6 , Figure 5 which is the overall implementation flowchart of the agent training provided in Embodiment 2 of this application, Figure 6 and which is the detailed process schematic diagram of the agent training provided in Embodiment 2 of this application. This process is divided into four parts. Specifically:

[0110] At the beginning of agent training, first parse the command-line parameters and obtain the hyperparameter configuration of the model, and set and save the agent directory.

[0111] Then, determine whether to start the rendering mode according to the environment rendering parameter in the model hyperparameter configuration. If the rendering parameter is true, start the rendering mode, convert it to the agent test mode, and ensure that the weights and parameters remain unchanged during the model test process; if the rendering parameter is false, do not start the rendering mode and keep the current agent training mode and parameters unchanged.

[0112] Next, create an instance of the agent training environment, set the number of random seeds, and obtain the number of actions of the agent from the action space for constructing the output layer of the multi-actor network.

[0113] Finally, determine whether the above agent directory has been created. If it does not exist, recreate the agent directory and save it in memory; if it exists, set a globally fixed seed value using a random number generator. Thus, the creation of the training environment is completed.

[0114] The second part is to construct and initialize the shared agent Share_Agent in the above training environment. The shared agent may include the network structures of the core agent Core_Agent and the custom optimizer Agent_Optimizer.

[0115] After the training environment is created, first initialize and construct the network structure of the core agent Core_Agent, including a feature encoder based on a gated recurrent unit, a multi-head self-attention network, a multi-actor network, and a multi-critic network.

[0116] Then construct a multi-network structure according to the number of actor networks and critic networks for data augmentation during the agent sampling process, including a multi-actor network structure and a multi-critic network structure.

[0117] Next, create a multi-network storage structure, build the network using linear layers and save it into the storage structure. The process is as follows: Determine whether the current layer number is less than the network layer number. If the current layer number is less than the network layer number, construct a linear layer as the input layer of the current actor or critic network, and then determine again whether the hidden layer number is less than the current layer number minus one; if the hidden layer number is less than the current layer number minus one, construct a linear layer as the intermediate hidden layer of the current actor or critic network, and then return to the above step of determining whether the current layer number is less than the network layer number. If the hidden layer number is greater than the current layer number minus one, construct a single linear layer as the output layer of the current actor or critic network and add it to the storage structure, and then return to the above step of determining whether the current layer number is less than the network layer number. Repeat this process until the current layer number reaches the network layer number and then proceed to the next step.

[0118] When the above current layer number is greater than the network layer number, construct and initialize the custom optimizer of the core agent, traverse the parameter groups of the optimizer, initialize specific shared memory states for the parameters in each parameter group, obtain the shared states, and configure the multi-threaded shared memory in the optimizer so that multiple processes can access and update the same state variables during the parallel training process. At this point, the initialization of the shared agent is completed.

[0119] The third part is to execute the parallel training process of the shared agent Share_agent in a multi-process parallel environment after the initialization of the shared agent is completed, and assign different tasks to different sub-processes to synchronize the status information of the local model configuration, and perform information sampling with the enhancement of the multi-head self-attention network and the multi-network structure.

[0120] After the initialization of the above shared agent is completed, first create instances of the local training environment for each sub-process in a loop and start the sub-processes for agent training.

[0121] Then copy the core agent Core_Agent as the local model, reset the local environment to obtain the initial local state information. And perform image preprocessing on the input image, and convert the obtained local state information into a state tensor as the input of the agent. At this time, first initialize the round data storage list for each round, and specify an internal loop with 1000 rounds. In each round, the agent will interact with the environment once to start training the agent.

[0122] The training process of the agent includes: first, judge whether the current round number is less than the training round number. If so, share the state information of the local model and the core agent, and reset the agent to obtain the hidden state. At this time, then judge whether the current round is less than 1000. If it is less than 1000, the core agent interacts with the external environment for information sampling, and calculates the value probability weight value output by the multi-critic network; then use the multi-head self-attention network to calculate the weights of multiple policies output by the multi-actor network to obtain the action decision, save the average reward and loss size of the agent training in this round, and finally return to execute the above step of judging whether the current round is less than 1000 until the current round reaches 1000. After the current round reaches 1000, calculate and save the cumulative discounted total reward and total loss size of the agent's current round of training. At this time, return to execute the above step of judging whether the current round number is less than the training round number.

[0123] Until the current round number reaches the training round number, complete the iterative training of the shared agent.

[0124] The fourth part is after completing the above iterative training, calculate the cumulative discounted total reward and total loss size of the agent, synchronize the gradients to the shared agent Share_agent for parameter update, and perform asynchronous training on the agent until the training ends.

[0125] After the parameters of the shared agent are updated, first end the process, create a list of subprocess objects, and add the subprocess objects to the list to facilitate subsequent management of the subprocesses.

[0126] Then start multiple independent subprocesses in a loop for asynchronous training of the agent. Each process shares the same model, optimizer, and parameters and is added to the subprocess list. During the asynchronous training process, first judge whether the current number is less than the number of processes. If so, perform subprocess parameter passing, start the process and perform asynchronous training of the agent, add the subprocess object to the list for management, and return to execute the above step of judging whether the current number is less than the number of processes until the current number reaches the number of processes. If not, directly wait for all processes to end, block the main process until the corresponding subprocesses complete their work, so as to ensure that the main process will wait for all subprocesses to execute before continuing to execute new tasks, and synchronize the end nodes of all processes. Finally, end the training process of the agent.

[0127] In this application, a gated recurrent unit is used as a key module of the state encoder in the agent, which can quickly and effectively capture the temporal dependencies in the state sequence, thereby improving the training speed and generalization ability of the model. At the same time, by implementing a parallel network structure to replace the traditional single-agent network, multiple actor-critic networks are used to generate diverse strategies in parallel; each critic network independently evaluates the policy effect of the corresponding actor-critic and outputs a value, and calculates the average value of multiple values, which can reduce the variance of value estimation and improve the accuracy of value estimation, thereby enhancing the adaptability and stability of the agent in a high-dimensional and complex exploration environment. On this basis, a decision optimization method combining multiple actor-critic networks and multi-head self-attention networks is also designed. By using multi-head self-attention to evaluate the weights between the policy outputs from multiple parallel actor-critics, the agent's ability to capture different policy features can be enhanced, the robustness of the model can be improved, and the overall decision-making process can be optimized.

[0128] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the agent training method of this application. Based on this technical concept, more forms of simple transformations are within the protection scope of this application.

[0129] This application also provides an agent training device. Please refer to Figure 7 , Figure 7 which is a schematic diagram of the module structure of the agent training device in the embodiment of this application; the agent training device includes: A training environment module 601, configured to create a training environment for the agent according to the model hyperparameter configuration. An agent module 602, configured to initialize a shared agent based on the training environment. The shared agent is provided with a network structure of a core agent and a custom optimizer, and the core agent is provided with a multi-head self-attention network and multiple actor-critic networks. An iterative training module 603, configured to iteratively train the shared agent according to the input image in a multi-process parallel environment to obtain the cumulative discounted total reward and the total loss size of the shared agent. A parameter update module 604, configured to update the parameters of the shared agent according to the cumulative discounted total reward and the total loss size.

[0130] As an implementation manner, the training environment module 601 is further configured to determine the rendering mode state according to the model hyperparameter configuration parsed from the program command line parameters; set the number of random seeds based on the agent training environment instance and obtain the number of actions from the action space; use a random number generator to set a globally fixed seed value according to the number of random seeds; and determine the training environment of the agent according to the rendering mode state, the number of actions, and the seed value.

[0131] As an implementation manner, the agent module 602 is further configured to initialize the network structure of the core agent. The core agent is provided with a feature encoder based on a gated recurrent unit, a multi-head self-attention network, a multi-actor network, and a multi-critic network. The multi-head self-attention network is used to optimize the multi-actor network; determine a multi-network structure according to the number of the multi-actor networks and the number of the multi-critic networks; based on the current layer number of the multi-network structure, construct a network using a linear layer and save it to a preset multi-network storage structure, where the multi-network storage structure is used to accommodate multiple networks of the shared agent; configure the shared memory state of a custom optimizer to complete the initialization of the shared agent.

[0132] As an implementation manner, the iterative training module 603 is further configured to, in a multi-process parallel environment, circularly create instances of the local training environment for each sub-process and start the instances of the local training environment for agent training; copy the core agent as a local model and determine local state information according to the local model and the input image; convert the local state information into a state tensor; perform iterative training on the shared agent according to the state tensor to obtain the cumulative discounted total reward and the total loss size of the shared agent.

[0133] As an implementation manner, the iterative training module 603 is further configured to perform training on the shared agent according to the state tensor; when the current round number is less than a preset training round number, share the state information of the local model and the core agent; determine the value probability weight value output by the multi-critic network according to the information sampling of the interaction between the core agent and the external environment; use the multi-head self-attention network to weight-process multiple policies output by the multi-actor network to obtain an action policy; until the current round number reaches the preset training round number, determine the cumulative discounted total reward and the total loss size of the shared agent according to the value probability weight value and the action policy.

[0134] As an implementation manner, the parameter update module 604 is further configured to, after the update is completed, end the current sub-process and create a sub-process list; when the current number is less than the current quantity, circularly start each sub-process to perform asynchronous training on the shared agent until all sub-processes in the sub-process list end; after all sub-processes in the sub-process list end, synchronize the end nodes of all sub-processes.

[0135] Other embodiments or specific implementation manners of the agent training device of the present application may refer to the above method embodiments and will not be elaborated here.

[0136] The intelligent agent training device provided by this application adopts the intelligent agent training method in the above embodiment, which can solve the technical problems that reinforcement learning is still prone to falling into local optimum and overfitting when the external exploration environment is sparse or the training data is insufficient during the application process, resulting in low generalization and low accuracy of the intelligent agent. Compared with the prior art, the beneficial effects of the intelligent agent training device provided by this application are the same as those of the intelligent agent training method provided by the above embodiment, and other technical features in the intelligent agent training device are the same as the features disclosed in the method of the above embodiment, which will not be elaborated here.

[0137] This application provides an intelligent agent training device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the intelligent agent training method in the first embodiment above.

[0138] Refer to the following Figure 8 , Figure 8 , which is a schematic structural diagram of a device for the hardware operating environment involved in the intelligent agent training method in an embodiment of this application, and shows a schematic structural diagram of an intelligent agent training device suitable for implementing the embodiment of this application. The intelligent agent training device in the embodiment of this application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 8 The shown intelligent agent training device is only an example and should not impose any limitations on the functions and usage scope of the embodiments of this application.

[0139] As Figure 8As shown, the agent training device may include a processing device 1001 (such as a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to the program stored in the read-only memory 1002 or the program loaded from the storage device 1003 into the random access memory 1004. In the random access memory 1004, various programs and data required for the operation of the agent training device are also stored. The processing device 1001, the read-only memory 1002, and the random access memory 1004 are connected to each other through a bus 1005. The input / output interface 1006 is also connected to the bus. Generally, the following systems may be connected to the input / output interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the agent training device to communicate with other devices wirelessly or wiredly to exchange data. Although the figure shows an agent training device with various systems, it should be understood that it is not required to implement or have all the shown systems. Instead, more or fewer systems may be implemented or had.

[0140] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program codes for executing the methods shown in the flowcharts. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device, or installed from the storage device 1003, or installed from the read-only memory 1002. When the computer program is executed by the processing device 1001, the above functions defined in the methods of the embodiments disclosed in the present application are executed.

[0141] The agent training device provided in the present application adopts the agent training method in the above embodiments, and can solve the technical problems that in the application process of reinforcement learning, when the external exploration environment is sparse or the training data is insufficient, it is easy to fall into a local optimum and cause overfitting, resulting in low generalization and low accuracy of the agent. Compared with the prior art, the beneficial effects of the agent training device provided in the present application are the same as the beneficial effects of the agent training method provided in the above embodiments, and other technical features in the agent training device are the same as the features disclosed in the method of the previous embodiment, and will not be elaborated here.

[0142] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in a suitable manner in any one or more embodiments or examples.

[0143] As described above, the above is only the specific implementation manner of this application, but the protection scope of this application is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in this application, and all of them should be covered by the protection scope of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

[0144] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., computer programs) stored thereon, and the computer-readable program instructions are used to execute the intelligent agent training method in the above embodiments.

[0145] The computer-readable storage medium provided by this application can be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination of the above. More specific examples of the computer-readable storage medium may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM: Random Access Memory), read-only memory (ROM: Read Only Memory), erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this embodiment, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium can be transmitted by any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0146] The above computer-readable storage medium can be included in the intelligent agent training device; or it can exist separately without being assembled into the intelligent agent training device.

[0147] The above computer-readable storage medium carries one or more programs, which, when executed by an agent training device, cause the agent training device to: create a training environment for an agent according to model hyperparameter configurations; initialize a shared agent based on the training environment, where the shared agent is provided with a network structure of a core agent and a custom optimizer, and the core agent is provided with a multi-head self-attention network and a multi-actor network; in a multi-process parallel environment, iteratively train the shared agent according to input images to obtain the cumulative discounted total reward and the total loss magnitude of the shared agent; update the parameters of the shared agent according to the cumulative discounted total reward and the total loss magnitude.

[0148] Computer program code for performing the operations of this application can be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., by using an Internet service provider to connect through the Internet).

[0149] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in the flowchart or block diagram can represent a module, a program segment, or a part of code that contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks can occur in a different order than that marked in the accompanying drawings. For example, two consecutive blocks shown can actually be executed substantially in parallel, and they can sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0150] The modules described in the embodiments of this application can be implemented in software or in hardware. In some cases, the name of the module does not constitute a limitation on the unit itself.

[0151] The readable storage medium provided in this application is a computer-readable storage medium. The computer-readable storage medium stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned agent training method, which can solve the technical problems that reinforcement learning is still prone to falling into local optimality and overfitting when the external exploration environment is sparse or the training data is insufficient during the application process, resulting in low generalization and accuracy of the agent. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the agent training method provided in the above embodiments, and will not be elaborated here.

[0152] This application also provides a computer program product, including a computer program, and when the computer program is executed by a processor, it implements the steps of the agent training method as described above.

[0153] The computer program product provided in this application can solve the technical problems that reinforcement learning is still prone to falling into local optimality and overfitting when the external exploration environment is sparse or the training data is insufficient during the application process, resulting in low generalization and accuracy of the agent. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the agent training method provided in the above embodiments, and will not be elaborated here.

[0154] The above are only some embodiments of this application, and do not limit the patent scope of this application. Any equivalent structural transformation made using the description and drawings of this application under the technical concept of this application, or any direct / indirect application in other related technical fields, is included in the patent protection scope of this application.

Claims

1. An agent training method, characterized in that, The method includes: Create a training environment for the agent according to the model hyperparameter configuration; Initialize a shared agent based on the training environment, where the shared agent is set with the network structures of a core agent and a custom optimizer, and the core agent is set with a multi-head self-attention network and a multi-actor-critic network; In a multi-process parallel environment, iteratively train the shared agent according to the input image to obtain the cumulative discounted total reward and the total loss size of the shared agent; Update the parameters of the shared agent according to the cumulative discounted total reward and the total loss size.

2. The method according to claim 1, characterized in that, The step of creating a training environment for the agent according to the model hyperparameter configuration includes: Determine the rendering mode status according to the model hyperparameter configuration parsed from the program command line parameters; Based on the agent training environment instance, set the number of random seeds and obtain the number of actions from the action space; According to the number of random seeds, use a random number generator to set a globally fixed seed value; Determine the training environment of the agent according to the rendering mode status, the number of actions, and the seed value.

3. The method according to claim 1, characterized in that, The step of initializing a shared agent based on the training environment includes: Initialize the network structure of the core agent, where the core agent is set with a feature encoder based on a gated recurrent unit, a multi-head self-attention network, a multi-actor-critic network, and a multi-critic network, and the multi-head self-attention network is used to optimize the multi-actor-critic network; Determine the multi-network structure according to the number of the multi-actor-critic networks and the number of the multi-critic networks; Based on the current layer number of the multi-network structure, use a linear layer to construct a network and save it to a preset multi-network storage structure, where the multi-network storage structure is used to accommodate multiple networks of the shared agent; Configure the shared memory status of the custom optimizer to complete the initialization of the shared agent.

4. The method according to claim 3, wherein The step of iteratively training the shared agent according to the input image in a multi-process parallel environment to obtain the cumulative discounted total reward and the total loss size of the shared agent includes: In a multi-process parallel environment, repeatedly create instances of the local training environment for each sub-process and start the instances of the local training environment for agent training; Copy the core agent as the local model and determine the local state information according to the local model and the input image; Convert the local state information into a state tensor; Iteratively train the shared agent according to the state tensor to obtain the cumulative discounted total reward and the total loss size of the shared agent.

5. The method according to claim 4, wherein The step of iteratively training the shared agent according to the state tensor to obtain the cumulative discounted total reward and the total loss size of the shared agent includes: Train the shared agent according to the state tensor; When the current round number is less than the preset training round number, share the state information of the local model and the core agent; Determine the value probability weight value output by the multi-critic network according to the information sampling of the interaction between the core agent and the external environment; Use the multi-head self-attention network to weight the multiple policies output by the multi-actor family network to obtain an action policy; Until the current round reaches the preset number of training rounds, determine the cumulative discounted total reward and the total loss size of the shared agent according to the value probability weight value and the action policy.

6. The method according to claim 4, wherein After the step of updating the parameters of the shared agent according to the cumulative discounted total reward and the total loss size, it further includes: After the update is completed, end the current subprocess and create a subprocess list; When the current number is less than the current quantity, loop to start each subprocess to perform asynchronous training on the shared agent until all subprocesses in the subprocess list end; After all subprocesses in the subprocess list end, synchronize the end nodes of all subprocesses.

7. An agent training device, characterized in that, The device includes: A training environment module for creating a training environment for the agent according to the model hyperparameter configuration; An agent module for initializing a shared agent based on the training environment, where the shared agent is provided with a network structure of a core agent and a custom optimizer, and the core agent is provided with a multi-head self-attention network and a multi-actor family network; An iterative training module for iteratively training the shared agent according to the input image in a multi-process parallel environment to obtain the cumulative discounted total reward and the total loss size of the shared agent; A parameter update module for updating the parameters of the shared agent according to the cumulative discounted total reward and the total loss size.

8. An agent training device, characterized in that, The device includes: a memory, a processor, and a computer program stored on the memory and executable on the processor, where the computer program is configured to implement the steps of the agent training method according to any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the agent training method according to any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes a computer program. When the computer program is executed by a processor, it implements the steps of the agent training method according to any one of claims 1 to 6.

Images