Intelligent agent training method, intelligent agent operation method and device

Abstract

The invention discloses an agent training method and device and an agent operation method and device, and relates to the technical field of artificial intelligence. A specific embodiment of the method comprises the following steps: when an agent is trained, training empirical data for training and shared information among a plurality of agents are combined to jointly train a set neural network of the agent, so that the agent generates an execution action through the set neural network, and the execution action is evaluated by using a target loss function; the target loss function in the set neural network provided by the embodiment of the invention is composed of the self loss function and the shared loss function among the multiple agents, so that the training efficiency of the neural network model in the multiple agents is improved, and the convergence efficiency of the network model is improved; and the excessive differentiation degree of the multiple agents is reduced, and the operation accuracy of the multiple agents is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an intelligent agent training method, an intelligent agent operation method and a device. Background Art

[0002] At present, artificial intelligence has been widely used in a variety of Internet applications. The reinforcement learning branch of artificial intelligence is usually used for multi-agent model training in a certain set environment. The set environment may include collaborative tasks of multiple agents, games of multiple agents, planning of multiple agents, group decision-making of multiple agents, etc.

[0003] The current methods for training intelligent agents include: 1) centralized training, that is, training multiple intelligent agents as a whole; 2) decentralized training, training multiple intelligent agents independently of each other; Method 1) When the number of intelligent agents is large, there is a problem of wasting computing resources due to the increase in data dimension; Method 2) When the number of intelligent agents is large and the environment is complex, there is a problem of poor training effect. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides an intelligent agent training method, an intelligent agent operation method and an intelligent agent operation device, which can combine the training experience data and the shared information between multiple intelligent agents to jointly train the set neural network of the intelligent agent when training the intelligent agent, so that the intelligent agent generates an execution action through the set neural network, and uses the target loss function to evaluate the execution action; the target loss function in the set neural network provided by the embodiment of the present invention is composed of its own loss function and the shared loss function between multiple intelligent agents, which improves the training efficiency of the neural network model in multiple intelligent agents, improves the convergence efficiency of the network model, reduces the degree of excessive differentiation of multiple intelligent agents, and improves the accuracy of the operation of multiple intelligent agents.

[0005] To achieve the above-mentioned purpose, according to one aspect of an embodiment of the present invention, there is provided an agent training method, characterized in that it includes: for each agent, performing the following operations to train the set neural network contained in the agent until a trained set neural network is obtained: obtaining training experience data for the current training cycle from the experience data set, training the set neural network based on the training experience data and the shared information, so that the agent generates an execution action through the set neural network; evaluating the execution action through the target loss function contained in the set neural network; wherein the target loss function includes the agent's own loss function and the shared loss function.

[0006] Optionally, the setting neural network includes an action neural network; and training the setting neural network based on the experience data set and the shared information so that the agent generates an execution action through the setting neural network includes: obtaining training environment data from the training experience data, inputting the training environment data into the action neural network, so that the action neural network calculates action scores of one or more execution actions associated with the training environment data based on the training environment data, and selecting an execution action corresponding to the maximum action score from multiple action scores.

[0007] Optionally, the setting neural network also includes an evaluation neural network; evaluating the execution action through the target loss function included in the setting neural network further includes: evaluating the execution action using the target loss function included in the evaluation neural network; updating the setting neural network according to the evaluation results of the evaluation neural network on one or more execution actions.

[0008] Optionally, the training of the setting neural network included in the intelligent agent also includes: iteratively performing the following operations until an iteration stop condition is met to obtain the trained setting neural network: obtaining a set number of training experience data from the training experience data set; using the training environment data included in the training experience data and the execution action associated with the training environment data to train the action neural network included in the setting neural network; based on the target action output by the action neural network and the next environment data for the execution action, training the evaluation neural network included in the setting neural network so that the evaluation neural network outputs an evaluation of the action neural network; and updating the model parameters of the action neural network and / or the evaluation neural network according to the evaluation results.

[0009] Optionally, the agent is provided with update conditions for the set neural network; the method further comprises: when the agent satisfies the update conditions, updating the shared information in the environmental data source and the experience data set of the agent to further train the set neural network.

[0010] Optionally, after the agent generates an execution action, it further includes: sending data of the execution action to the environmental data source to update the experience data contained in the environmental data source and the shared information between multiple agents; using the updated shared information to further train the set neural network in each agent associated with the environmental data source.

[0011] Optionally, the intelligent agent training method further includes: for each execution action of any intelligent agent, performing the following operations: generating current environment data corresponding to the execution action, action data of the execution action, reward data obtained for the execution action, and environment data corresponding to the next execution action after the execution action as a set of experience data into the experience data set of the intelligent agent.

[0012] Optionally, the agent training method further includes: constructing a self-loss function corresponding to the agent based on the calculation relationship between the experience data contained in the experience data set in the agent and the set reference reward data; and / or constructing a distance calculation relationship between the probability distribution of the evaluation neural network output and the probability distribution of shared information; based on the calculation relationship and the shared information between multiple agents, constructing a shared loss function.

[0013] To achieve the above-mentioned purpose, according to the second aspect of an embodiment of the present invention, there is provided an intelligent agent operation method, comprising: obtaining current environmental data matching the intelligent agent from an environmental data source; generating an execution action of the intelligent agent based on the current environmental data through a set network model contained in the intelligent agent; wherein the set network model is trained by the intelligent agent training method described in the first aspect.

[0014] To achieve the above-mentioned purpose, according to a third aspect of an embodiment of the present invention, there is provided an agent training device, characterized in that it comprises: a data acquisition module and a model training module; wherein:

[0015] The data acquisition module is used to acquire, in response to training multiple intelligent agents, a set of experience data used to train the intelligent agents and shared information between the multiple intelligent agents from an environmental data source;

[0016] The model training module is used to perform the following operations to train the intelligent agent for each iterative training cycle: obtaining training experience data for the current iterative cycle from the experience data set, training the set neural network contained in the intelligent agent based on the training experience data and the shared information, so that the intelligent agent generates an execution action through the set neural network; and evaluating the execution action through the target loss function contained in the set neural network; the target loss function includes the intelligent agent's own loss function and the shared loss function.

[0017] Optionally, the setting neural network included in the intelligent agent training device includes an action neural network; and the training of the setting neural network based on the experience data set and the shared information so that the intelligent agent generates an execution action through the setting neural network includes: obtaining training environment data from the training experience data, inputting the training environment data into the action neural network, so that the action neural network calculates action scores of one or more execution actions associated with the training environment data based on the training environment data, and selects an execution action corresponding to the maximum action score from multiple action scores.

[0018] Optionally, the setting neural network included in the intelligent agent training device also includes an evaluation neural network; evaluating the execution action through the target loss function included in the setting neural network further includes: evaluating the execution action using the target loss function included in the evaluation neural network; updating the setting neural network according to the evaluation results of the evaluation neural network on one or more execution actions.

[0019] Optionally, the agent training device is used to train the set neural network included in the agent, and also includes: iteratively performing the following operations until the iteration stop condition is met to obtain the trained set neural network: obtaining a set number of training experience data from the training experience data set; using the training environment data included in the training experience data and the execution action associated with the training environment data to train the action neural network included in the set neural network; based on the target action output by the action neural network and the next environment data for the execution action, training the evaluation neural network included in the set neural network so that the evaluation neural network outputs an evaluation of the action neural network; updating the model parameters of the action neural network and / or the evaluation neural network according to the evaluation results.

[0020] Optionally, the agent included in the agent training device is set with update conditions for the set neural network; the method also includes: when the agent meets the update conditions, updating the shared information in the environmental data source and the experience data set of the agent to further train the set neural network.

[0021] Optionally, the intelligent agent training device is used to further include, after the intelligent agent generates an execution action, sending data of the execution action to the environmental data source to update the experience data contained in the environmental data source and the shared information between multiple intelligent agents; and using the updated shared information to further train the set neural network in each intelligent agent associated with the environmental data source.

[0022] Optionally, the intelligent agent training device is further used to perform the following operations for each execution action of any intelligent agent: storing the current environment data corresponding to the execution action, the action data of the execution action, the reward data obtained for the execution action, and the environment data corresponding to the next execution action after the execution action as a set of experience data into the experience data set of the intelligent agent.

[0023] Optionally, the intelligent agent training device is further used to construct a self-loss function corresponding to the intelligent agent based on the calculation relationship between the experience data contained in the experience data set in the intelligent agent and the set reference reward data; and / or, to construct a distance calculation relationship between the probability distribution of the evaluation neural network output and the probability distribution of shared information; based on the calculation relationship and the shared information between multiple intelligent agents, to construct a shared loss function.

[0024] To achieve the above-mentioned purpose, according to the fourth aspect of an embodiment of the present invention, there is provided an electronic device for intelligent agent operation, characterized in that it includes: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of the above-mentioned intelligent agent training methods and intelligent agent operation methods.

[0025] To achieve the above-mentioned purpose, according to the fifth aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored, characterized in that when the program is executed by a processor, a method as described in any one of the above-mentioned intelligent agent training methods and intelligent agent operation methods is implemented.

[0026] An embodiment of the above invention has the following advantages or beneficial effects: when training an intelligent agent, the set neural network of the intelligent agent is jointly trained in combination with the training experience data and the shared information between multiple intelligent agents, so that the intelligent agent generates an execution action through the set neural network, and uses the target loss function to evaluate the execution action; the target loss function in the set neural network provided by the embodiment of the present invention is composed of its own loss function and the shared loss function between multiple intelligent agents, which improves the training efficiency of the neural network model in multiple intelligent agents, improves the convergence efficiency of the network model, reduces the degree of excessive differentiation of multiple intelligent agents, and improves the accuracy of the operation of multiple intelligent agents.

[0027] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0029] Figure 1 is a flow chart of an agent training method provided by an embodiment of the present invention;

[0030] Figure 2 It is a schematic diagram of a structure of multiple intelligent agents running according to an embodiment of the present invention;

[0031] Figure 3 It is a flow chart of a training setting network model provided by an embodiment of the present invention;

[0032] Figure 3A A schematic diagram of an existing method for training an intelligent agent provided by an embodiment of the present invention;

[0033] Figure 3B A schematic diagram of another existing method for training an intelligent agent provided by an embodiment of the present invention;

[0034] Figure 3C A schematic diagram of an agent training method provided by one embodiment of the present invention;

[0035] Figure 4 A schematic diagram of a flow chart of an intelligent agent operation method provided by an embodiment of the present invention;

[0036] Figure 5 is a schematic diagram of the structure of an intelligent agent training device provided by one embodiment of the present invention;

[0037] Figure 6 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0038] Figure 7 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0040] It should be noted that the collection, collection, updating, analysis, processing, use, transmission, storage and other aspects of user personal information involved in the technical solution of this disclosure are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain the security of user personal information, network security and national security.

[0041] like Figure 1 As shown, an embodiment of the present invention provides an agent training method, which may include the following steps:

[0042] Step S101: In response to training multiple intelligent agents, an experience data set used for training the intelligent agents and shared information among the multiple intelligent agents are obtained from an environmental data source.

[0043] Specifically, multi-agent reinforcement learning is currently widely used in a large number of complex scenarios, such as collaborative control (multiple agents achieve common goals through collaborative control, such as drone collaborative tasks, robot collaborative assembly, etc.), adversarial games (adversarial games between multiple agents, such as Go, etc.), path planning (multiple agents perform path planning in a limited space, such as traffic flow control, unmanned driving, etc.), multi-agent communication (information interaction and sharing between multiple agents, such as social networks, smart cities, etc.), and group decision-making (multiple agents make decisions in a common environment, etc.).

[0044] In an embodiment of the present invention, the environment used for the operation and training of the agent is the application scenario to which the agent belongs, such as the application scenario for a maze game, where the environment is a maze and the agent is the object walking through the maze. The environmental data contained in the environment is set by the specific application scenario. For example, in a maze game, the environmental data may include data such as the location of facilities in the maze and whether it is possible to walk, while in a football game, the environmental data may include data such as the location of the ball, the location of teammates, and the location of boundaries. In an embodiment of the present invention, the environmental data may be stored in an environmental data source, and the present invention does not limit the specific form and content of the environmental data source.

[0045] Specifically, in response to training multiple agents, an experience data set used to train the agents and shared information between the multiple agents are obtained from an environmental data source. The experience data set may be a set of historical experience data of the agents, so that in each training cycle, a set amount of experience data may be extracted from the set of historical experience data as training data;

[0046] Furthermore, in an embodiment of the present invention, the training data also includes shared information between multiple agents, wherein the shared information can be determined by a collaborative mechanism, and the efficiency and accuracy of the convergence of the agent model can be improved by sharing the information.

[0047] It is understandable that the content of shared information is different for different application scenarios. For example, for a football game application scenario, shared information may include the environmental data received by each player. It is understandable that by using shared information to train the network model of the intelligent agent, the globality of the training intelligent agent is improved, thereby improving the training effect of the intelligent agent. Further, for example, for a scenario where multiple intelligent agents collaborate, a common training goal for multiple intelligent agents can be set based on shared information, and intelligent agents in different states have the same evaluation of the same action based on the shared information. For a scenario where multiple intelligent agents compete, different training goals can be set for different intelligent agents based on shared information. The present invention does not limit specific application scenarios or specific data content of shared information.

[0048] Step S102: For each iterative training cycle, perform the following operations for training any agent: obtain the training experience data for the current iterative cycle from the experience data set, train the set neural network contained in the agent based on the training experience data and the shared information, so that the agent generates an execution action through the set neural network; evaluate the execution action through the target loss function contained in the set neural network; wherein the target loss function includes the agent's own loss function and the shared loss function.

[0049] Specifically, step S102 describes the process of setting a neural network to generate an execution action and using its loss function to evaluate its action included in the training of the intelligent agent. In the embodiment of the present invention, the execution actions of the intelligent agent in different application scenarios (environments) are different. For example, for each intelligent agent in a maze game scene, its execution action can be forward, backward, turn left, turn right, turn back, etc.; for each intelligent agent in a football game scene, its execution action can be forward, backward, left, right, back, kick the ball, defend, etc. The present invention does not limit the specific application scenario and the specific execution action of the intelligent agent.

[0050] The setting neural network contained in each intelligent agent is composed of an action neural network (actor network) and an evaluation neural network (critic network). Therefore, the setting neural network contained in the training intelligent agent is the training action neural network and the evaluation neural network. The action neural network is used to generate execution actions, and the evaluation neural network is used to evaluate the execution actions. The trained setting neural network is obtained through iterative training.

[0051] Specifically, training the action neural network based on the training experience data includes the training action neural network calculating the action scores of one or more execution actions associated with the current environment data (i.e., the training data) based on the environment in the training experience data, and selecting the execution action corresponding to the maximum action score from the multiple action scores. The experience data may be a data group, which includes the current environment data corresponding to the execution action to be generated, the action data of the execution action, the reward data obtained for the execution action, and the environment data corresponding to the next execution action after the execution action;

[0052] That is, the setting neural network includes an action neural network; the setting neural network included in the training of the agent based on the experience data set and the shared information so that the agent generates an execution action through the setting neural network includes: obtaining environmental data in the training experience data, inputting the environmental data into the action neural network, so that the action neural network calculates the action scores of one or more execution actions associated with the environmental data based on the environmental data, and selects the execution action corresponding to the maximum action score from multiple action scores.

[0053] Furthermore, the execution action is evaluated using the target loss function included in the evaluation neural network, that is, the setting neural network also includes an evaluation neural network; evaluating the execution action using the target loss function included in the setting neural network further includes: evaluating the execution action using the target loss function included in the evaluation neural network; and updating the setting neural network based on the evaluation results of the evaluation neural network on one or more execution actions.

[0054] Furthermore, in an embodiment of the present invention, the environment data source can store the experience data sent by each agent in the environment (for example: the current environment data of the agent, the execution action generated based on the current environment data, the reward data obtained for the execution action, the next environment data, etc.), and store the experience data of the agent in the experience data set accordingly. That is, for each execution action of any agent, the following operation is performed: the current environment data corresponding to the execution action, the action data of the execution action, the reward data obtained for the execution action, and the environment data corresponding to the next execution action after the execution action are stored as a set of experience data in the experience data set. Among them, the reward data is a data representation of the evaluation neural network evaluating the execution action.

[0055] Furthermore, in an embodiment of the present invention, shared information can be extracted from environmental data sources through a collaborative mechanism, thereby providing guidance for the model training of the evaluation neural network (critic network) of each agent, so as to improve the efficiency and accuracy of the agent model convergence. It can be understood that the content of the shared information is different for different application scenarios. For example, for a scenario where multiple agents collaborate, the shared information can be a common training target for multiple agents, and agents in different states based on the shared information have the same evaluation of the same action; and for a scenario where multiple agents compete, the shared information has different training targets for different agents. The present invention does not limit the specific application scenarios or the specific data content of the shared information.

[0056] Furthermore, after the agent generates an execution action, the data of the execution action is sent to the environmental data source to update the experience data contained in the environmental data source and the shared information between multiple agents; the updated shared information is used to further train the set neural network in each agent associated with the environmental data source; it can be understood that the shared information in the environmental data source is dynamically updated. During the training process, each time the agent calculates and generates an execution action, it can send data related to the execution action to the environmental data source to update the environmental data and the shared information between multiple agents, thereby further training the set neural network in each agent associated with the environmental data source based on the updated data (environmental data and shared information); it can be understood that an environmental data source can be set for each environment (i.e., application scenario), and the present invention does not limit the specific form and storage location of the environmental data source.

[0057] like Figure 2 As shown, an embodiment of the present invention provides a schematic diagram of the structure of multiple agents running: wherein Env represents the data contained in the environment data source (including the environment data of the agent, the shared information of the agent, etc.); i Represents the current environment data of one of the multiple agents i obtained from the environment management terminal; A i Represents the execution action generated by agent i; Actor represents the action neural network in the set neural network contained in the agent, Critic represents the evaluation neural network in the set neural network contained in the agent; Collaboration mechanism represents the collaboration mechanism used by the environment management end, which is used to generate shared information based on the experience data received from each agent.

[0058] Among them, the Actor network (i.e., the action neural network) generates an execution action based on the current environment data, and the Critic network (i.e., the evaluation neural network) uses the current environment data as the input of its loss function, thereby evaluating the output of the Actor through the output of the loss function; wherein, the Critic's loss function consists of two parts, one is the self-loss function of the setting neural network of the agent, and the other is the shared information loss function; that is, the setting neural network also includes an evaluation neural network; the evaluation neural network includes the self-loss function and the shared information loss function of the agent; further includes: using the self-loss function contained in the evaluation neural network in combination with the shared information loss function to evaluate the execution action; updating the setting neural network according to the evaluation results of the evaluation neural network on one or more execution actions.

[0059] It can be seen that the embodiments of the present invention improve the accuracy and training effect of the set neural network model contained in the training agent by combining the loss function of the neural network set by the agent itself and the shared information loss function, thereby improving the model convergence efficiency.

[0060] like Figure 3 As shown, an embodiment of the present invention provides a schematic diagram of a process of setting a network model for training an intelligent agent, and the process may include the following steps:

[0061] Specifically, steps S301 to S306 describe the process of setting the neural network included in the iterative training of each agent in the embodiment of the present invention; for each agent, the training data obtained includes its own experience data set and shared information obtained from the environment management end, wherein a batch of data can be extracted from its own experience data set as training data; in the embodiment of the present invention, the agent and the environment service end can be connected to perform data transmission using any data transmission protocol. That is, for each agent connected to the environment management end, the following operations are performed: the shared information between multiple agents is obtained from the environment management end; the set neural network is trained using the experience data set stored by the agent and the shared information obtained.

[0062] The description of step S301 to step S306 is that the training of the setting neural network included in the intelligent agent also includes: iteratively performing the following operations until the iteration stop condition is met to obtain the trained setting neural network: obtaining a set number of training experience data from the training experience data set; using the training environment data included in the training experience data and the execution action associated with the training environment data to train the action neural network included in the setting neural network; based on the target action output by the action neural network and the next environment data for the execution action, training the evaluation neural network included in the setting neural network so that the evaluation neural network outputs an evaluation of the action neural network; updating the model parameters of the action neural network and / or the evaluation neural network according to the evaluation results.

[0063] The following is a comparative description of the method for training an intelligent agent provided by an embodiment of the present invention in conjunction with a schematic diagram of an existing method for training an intelligent agent;

[0064] Figure 3A An existing method for training an intelligent agent is shown, such as Figure 3A As shown, this method is to conduct centralized training for multiple agents, that is, multiple agents (Agent1~AgentI) are trained uniformly as one agent; when the number of agents is large, this method has the problem of wasting computing resources due to the increase in data dimension.

[0065] Figure 3B Another existing method for training an intelligent agent is shown, such as Figure 3B As shown, this method is to conduct decentralized training for multiple agents, that is, to train multiple agents (Agent1~Agent I) independently from each other; when the number of agents is large and the environment is complex, this method has the problem of poor training effect and high degree of differentiation among the agents.

[0066] Figure 3C An agent training method provided by an embodiment of the present invention is shown; Agent1~AgentI represent multiple agents, o i ,a i They represent the environmental data obtained by agent i and the associated execution actions generated; Environment represents the summary of the experience data set of each agent in the environment, where Figure 3C The Environment rectangle in the figure is for illustration only. Each agent has its own set of experience data. Information represents the shared information of multiple agents.

[0067] Specifically, Figure 3CThe method shown is described in steps S301 to S306 as follows: Step S301: Obtain a set amount of training experience data from a training experience data set.

[0068] Step S302: using the training environment data included in the training experience data and the execution actions associated with the training environment data to train the action neural network included in the set neural network.

[0069] In an embodiment of the present invention, DDPG (deep deterministic policy gradient) is used as a training algorithm for reinforcement learning of an agent. In a distributed DDPG algorithm, each agent executes the DDPG algorithm separately.

[0070] For example, the gradient of the actor network (action neural network) for training the i-th agent is shown in formula (1):

[0071]

[0072] Where D represents the buffer for storing training experience data, o i ,a i represent the data sampled from the buffer (i.e., the training environment data contained in the training experience data, and the execution action associated with the training environment data). i is the Q function of agent i (used to calculate the action score corresponding to the input observation o and action a), E represents the operator for calculating the statistical value, and u represents the actor network.

[0073] Step S303: Based on the target action output by the action neural network and the next environmental data for the execution action, the evaluation neural network included in the setting neural network is trained so that the evaluation neural network outputs an evaluation for the action neural network.

[0074] Specifically, the evaluation neural network included in the neural network is set to have a target loss function, and its target loss function is shown in formula (2):

[0075]

[0076] Among them, L c (θ i ) represents the loss function of the ith agent, L T (θ i) represents the shared loss function, represents the shared data learned by agent i from the shared information managed by the collaborative mechanism, and δ is the weight parameter. By combining the two parts of the loss function training model, the agent can obtain more shared information during the training process, which makes the agent have better generalization ability, thereby improving the ability to manage partial observability and environmental non-stationarity.

[0077] Furthermore, L c (θ i ) is calculated as shown in formula (3):

[0078]

[0079] in y represents the expected reward data (i.e., the set reference reward data); E represents the operator. The expression on the right side of the equal sign in formula (3) is the calculation relationship between the experience data contained in the experience data set of the agent and the set reference reward data, that is, the calculation relationship between the experience data contained in the experience data set of the agent and the set reference reward data is used to construct the loss function corresponding to the agent itself; oi' represents the next environment data for the execution of the action. For the i-th agent, its experience data can be represented by a set of <o i ,a i ,r i ,o i > represents; wherein, oi, ai represent the current environment data, the execution action associated with the current environment data, ri represents the reward data, and oi' represents the next environment data of the execution action.

[0080] That is, based on the calculation relationship between the experience data contained in the experience data set in the agent and the set reference reward data, a loss function corresponding to the agent itself is constructed; and / or, a distance calculation relationship between the probability distribution of the evaluation neural network output and the probability distribution of shared information is constructed; based on the calculation relationship and the shared information between multiple agents, a shared loss function is constructed.

[0081] Furthermore, the shared loss function L T (θ i ) is shown in formula (5):

[0082]

[0083] Formula (5) includes the distance calculation relationship between the constructed probability distribution of the evaluation neural network output and the probability distribution of the shared information; there are many ways to calculate the distance (such as KL divergence, JS distance, MMD distance, etc.), and the embodiment of the present invention takes KL divergence as an example, as shown in formula (5), wherein m is the number of experience data obtained from the experience data buffer, and T is the distillation temperature parameter (used to adjust the "smoothness" of the output).

[0084] Where q = softmax(z s / T), p = softmax(z t / T)

[0085] Zs represents the vector output of the last layer of the critic network, and Zt represents the shared information. Z is the feature matrix, αi is the coefficient of the ith agent, and zt is a linear combination of the logistic regression of multiple agent critic networks to determine the shared information; optionally, other methods (such as statistical methods such as weighted average method) can also be used to generate shared information.

[0086] Step S304: Update the model parameters of the action neural network and / or the evaluation neural network according to the evaluation result. Specifically, the evaluation result can be obtained by formula (2).

[0087] Step S305: Determine whether the iterative stopping condition of the training is met. If so, execute the step; otherwise, execute step S301 to perform cyclic iterative model training.

[0088] Step S306: Obtain a trained setting neural network.

[0089] Specifically, during the iterative training process, the model parameters of the action neural network and / or the evaluation neural network may be modified to achieve the effect of optimizing the model during training.

[0090] Furthermore, the agent is provided with update conditions for the set neural network; the method further comprises: when the agent satisfies the update conditions, further training the set neural network using the shared information updated by the environment management terminal and the set of experience data updated in the agent. The update conditions for setting the neural network may be setting N iterative training cycles, or an episode (a round, a game, etc.) in an application scenario. When each update condition for setting the neural network is met, the model parameters are updated, and the training for the next iterative cycle, that is, the shared information updated by the environment management terminal and the set of experience data updated in the agent are used to further train the set neural network until the condition for stopping the iteration is met. The condition for stopping the iteration may be that the trained set neural network model reaches the optimal convergence degree, or the trained model parameters no longer change.

[0091] The embodiments of the present invention achieve the beneficial effects of saving computing resources, accelerating the training of multi-agent algorithms, and improving the performance of multi-agent algorithms by combining the use of self-experience data and shared information to train intelligent agents, and by setting the update conditions of the neural network, while improving the global effect of model training and overcoming the problem of excessive differentiation of intelligent agents in the system. It can be understood that the multiple intelligent agents in the embodiments of the present invention can perform training iteratively during the operation process, so that there is no sequential dependency between the operation of the intelligent agents and the training of the intelligent agent models, and they can be performed sequentially or crosswise, which improves the flexibility of training and the efficiency of model training in the intelligent agents.

[0092] like Figure 4 As shown, an embodiment of the present invention provides a flow chart of an intelligent agent operation method, and the flow chart may include the following steps:

[0093] Step S401: Acquire current environment data matching the agent from the environment data source;

[0094] Step S402: Generate a corresponding execution action based on the current environment data through its trained setting network model; wherein the setting network model is trained by the intelligent agent training method provided by an embodiment of the present invention.

[0095] Specifically, the intelligent agent can use the setting network model trained by the intelligent agent training method of the embodiment of the present invention to generate corresponding execution actions based on the current environment data through the trained setting network model; calculate the action scores of one or more execution actions associated with the current environment data based on the current environment data through the action neural network in the setting network model, and select the execution action corresponding to the maximum value of the action score from multiple action scores.

[0096] It is understandable that, since different agents have different states in the same environment, each agent needs to obtain environmental data that matches itself (i.e., current environmental data) for its operation, that is, before determining each execution action. Still taking the maze game as an example, for a certain agent in the maze, local observation can be used to obtain the perceived current environmental data (local observation data within a set range), so that the agent can obtain the current environmental data (including facility data of the environment itself, and data such as the state of other surrounding agents), so that the agent can calculate the corresponding action based on the current environmental data, that is, obtain the current environmental data that matches the agent from the environment management terminal, and generate the corresponding execution action based on the current environmental data through its trained set network model.

[0097] like Figure 5 As shown, an embodiment of the present invention provides an intelligent agent training device 500, including: a data acquisition module 501 and a model training module 502; wherein,

[0098] The data acquisition module 501 is used to acquire, in response to training multiple intelligent agents, a set of experience data used to train the intelligent agents and shared information between the multiple intelligent agents from an environmental data source;

[0099] The model training module 502 is used to perform the following operations for training the set neural network contained in the agent for each agent until a trained set neural network is obtained: obtaining training experience data for the current training cycle from the experience data set, training the set neural network based on the training experience data and the shared information, so that the agent generates an execution action through the set neural network; evaluating the execution action through the target loss function contained in the set neural network; wherein the target loss function includes the agent's own loss function and the shared loss function.

[0100] An embodiment of the present invention also provides an electronic device running an intelligent agent, comprising: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in any of the above embodiments.

[0101] An embodiment of the present invention further provides a computer-readable medium on which a computer program is stored. When the program is executed by a processor, the method provided in any of the above embodiments is implemented.

[0102] Figure 6 An exemplary system architecture 600 is shown to which the method for agent operation or the apparatus for agent operation according to the embodiment of the present invention can be applied.

[0103] like Figure 6 As shown, system architecture 600 may include terminal devices 601, 602, 603, network 604 and server 605. Network 604 is used to provide a medium for communication links between terminal devices 601, 602, 603 and server 605. Network 604 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0104] In the embodiments of the present invention, the terminal device may be various types of intelligent terminal (eg, unmanned transportation equipment, robot, etc.).

[0105] The user can use the terminal devices 601, 602, 603 to interact with the server 606 through the network 604 to receive or send messages, etc. Various client applications can be installed on the terminal devices 601, 602, 603, such as client applications that execute intelligent agent functions, etc.

[0106] The terminal devices 601 , 602 , and 603 may be various electronic devices having a display screen and supporting various client applications, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0107] The server 606 may be a server that provides various services (e.g., a server where an environmental data source is located), such as a background management server that provides support for client applications used by users using terminal devices 601, 602, and 603. The background management server may process the received experience data request of the intelligent agent and feed back various experience data required for the intelligent agent to the terminal device.

[0108] It should be noted that the intelligent agent training method or intelligent agent operation method provided in the embodiment of the present invention is generally executed by the terminal devices 601, 602, and 603; accordingly, the intelligent agent training device is generally arranged in the terminal devices 601, 602, and 603.

[0109] It should be understood that Figure 6 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0110] Reference below Figure 6 , which shows a schematic diagram of the structure of a computer system 700 of a terminal device suitable for implementing an embodiment of the present invention. Figure 6 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0111] like Figure 7As shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage part 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the system 700 are also stored. The CPU 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0112] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage section 708 as needed.

[0113] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 709, and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 701, the above-mentioned functions defined in the system of the present invention are executed.

[0114] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0115] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0116] The modules and / or units involved in the embodiments of the present invention may be implemented in software or hardware. The modules and / or units described may also be provided in a processor, for example, they may be described as: a processor including a data acquisition module and a model training module. The names of these modules do not, in some cases, constitute limitations on the modules themselves, for example, the data acquisition module may also be described as a "module for acquiring training data used by training an intelligent agent from an environmental data source".

[0117] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently and not be assembled into the device. The above computer-readable medium carries one or more programs. When the above one or more programs are executed by a device, the device includes: in response to training multiple agents, obtaining the experience data set used for training the agents and the shared information between the multiple agents from the environmental data source; for each agent, performing the following operations of training the set neural network contained in the agent until a trained set neural network is obtained: obtaining the training experience data of the current training cycle from the experience data set, training the set neural network based on the training experience data and the shared information, so that the agent generates an execution action through the set neural network; evaluating the execution action through the target loss function contained in the set neural network; wherein the target loss function includes the self-loss function of the agent and the shared loss function.

[0118] The embodiments of the present invention can, when training an intelligent agent, combine training experience data and shared information between multiple intelligent agents to jointly train the set neural network of the intelligent agent, so that the intelligent agent generates execution actions through the set neural network and evaluates the execution actions using the target loss function; the target loss function in the set neural network provided by the embodiments of the present invention is composed of its own loss function and the shared loss function between multiple intelligent agents, which improves the training efficiency of the neural network model in multiple intelligent agents, improves the convergence efficiency of the network model, reduces the degree of excessive differentiation of multiple intelligent agents, and improves the accuracy of the operation of multiple intelligent agents.

[0119] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for training an intelligent agent, characterized in that: include: In response to training a plurality of intelligent agents, acquiring from an environmental data source a set of experience data used to train the intelligent agents and shared information between the plurality of intelligent agents; For each agent, perform the following operations to train the set neural network contained in the agent until a trained set neural network is obtained: Acquire training experience data of a current training cycle from the experience data set, and train the set neural network based on the training experience data and the shared information, so that the agent generates an execution action through the set neural network; The execution action is evaluated by the target loss function included in the set neural network; wherein the target loss function includes the agent's own loss function and the shared loss function.

2. The method according to claim 1, characterized in that The setting neural network includes an action neural network; The step of training the set neural network based on the experience data set and the shared information so that the agent generates an execution action through the set neural network includes: Acquire training environment data from the training experience data, and input the training environment data into the action neural network so that the action neural network calculates action scores of one or more execution actions associated with the training environment data based on the training environment data, and selects an execution action corresponding to the maximum value of the action score from multiple action scores.

3. The method according to claim 1, characterized in that The setting neural network also includes an evaluation neural network; Evaluating the execution action by the target loss function included in the setting neural network further includes: Using the target loss function included in the evaluation neural network, the execution action is evaluated; The setting neural network is updated according to the evaluation result of the evaluation neural network on one or more execution actions.

4. The method according to claim 2 or 3, characterized in that: The training of the agent includes setting a neural network, and further includes: Iteratively perform the following operations until the iteration stop condition is met to obtain the trained set neural network: Acquire a set amount of training experience data from the training experience data set; Using the training environment data included in the training experience data and the execution action associated with the training environment data to train the action neural network included in the setting neural network; Based on the target action output by the action neural network and the next environmental data for the execution action, training the evaluation neural network included in the setting neural network so that the evaluation neural network outputs an evaluation for the action neural network; The model parameters of the action neural network and / or the evaluation neural network are updated according to the evaluation results.

5. The method according to claim 4, characterized in that The intelligent agent is provided with update conditions for the set neural network; The method further includes: when the agent satisfies the update condition, updating the shared information in the environmental data source and the experience data set of the agent to further train the set neural network.

6. The method according to claim 1, characterized in that After the agent generates an execution action, the method further comprises: Sending the data of the execution action to the environmental data source to update the experience data contained in the environmental data source and the shared information among multiple agents; The updated shared information is used to further train the set neural network in each agent associated with the environmental data source.

7. The method according to claim 1, characterized in that Further including: For each action taken by any agent, perform the following operations: The current environment data corresponding to the execution action, the action data of the execution action, the reward data obtained for the execution action, and the environment data corresponding to the next execution action after the execution action are stored as a set of experience data in the experience data set of the intelligent agent.

8. The method according to claim 1, characterized in that: Further including: Constructing a self-loss function corresponding to the intelligent agent according to a calculation relationship between the experience data contained in the experience data set in the intelligent agent and the set reference reward data; and / or, A distance calculation relationship between the constructed probability distribution of the evaluation neural network output and the probability distribution of the shared information; Based on the calculation relationship and the shared information between multiple agents, a shared loss function is constructed.

9. An intelligent agent operation method, characterized in that: include: Obtain current environment data matching the agent from the environment data source; The set network model trained by the intelligent agent generates a corresponding execution action based on the current environment data; wherein the set network model is trained by the intelligent agent training method described in any one of claims 1-8.

10. An intelligent agent training device, characterized in that: include: Get the data module and model training module; among them, The data acquisition module is used to acquire, in response to training multiple intelligent agents, a set of experience data used to train the intelligent agents and shared information between the multiple intelligent agents from an environmental data source; The model training module is used to perform the following operations for training the set neural network contained in the agent for each intelligent agent until a trained set neural network is obtained: obtaining training experience data for the current training cycle from the experience data set, training the set neural network based on the training experience data and the shared information, so that the intelligent agent generates an execution action through the set neural network; evaluating the execution action through the target loss function contained in the set neural network; wherein the target loss function includes the self-loss function of the intelligent agent and the shared loss function.

11. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 9.

12. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images